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Preface 


The  purpose  of  this  research  was  to  design  and  integrate 
a  visual  servo  control  scheme  for  a  PUMA  560  robot  arm  that 
derives  its  position  feedback  information  from  a  Machine 
Intelligence  Corporation  (MIC)  vision  system.  The  vision 
system’s  television  camera  was  mounted  to  the  PUMA’s  third 
joint.  The  integrated  vision-robot  system  (VRS)  successfully 
employed  closed  loop,  static  and  dynamic,  visual  servo 
control  techniques  for  demonstrating  the  feasibility  of  using 
a  VRS  to  ground  refuel  an  aircraft. 

I  wish  to  express  my  deepest  gratitude  to  a  number  of 
people  who  helped  me  complete  this  exciting  research  project. 
In  particular  Dr.  Matthew  Kabrisky,  my  advisor,  and  Dr. 
Michael  Leahy  both  of  whom  provided  valuable  suggestions  and 
recommendations.  Dr.  Leahy’s  expertise  in  robot  control  and 
the  PUMA  560  was  invaluable.  This  research  effort  would  not 
have  been  a  "success"  without  his  help  and  assistance.  I  am 
forever  grateful  to  Dr.  Kabrisky  who  was  always  excited  to 
come  "underground"  to  view  my  progress  ("YOU  HAVE  20  SECONDS 
TO  DROP  YOUR  WEAPON",  to  quote  the  well-renowned  movie 
production  "RoboCop" ) .  Dr.  "K"  provided  unmeasurable 
guidance  and  direction,  especially  in  my  write-up.  I  would 
also  like  to  express  by  deepest  thanks  to  Mr.  Bob  Ewing 
(THANKS  BOB)  for  his  encouragement  and  excitement,  especially 
at  the  start  of  this  project.  Thanks  also  go  out  to  Lt  Col 
Zdzislaw  H.  Lewantowicz  for  his  efforts  in  reviewing  my  work. 
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facing  the  Air  Force.  A  special  thanks  go  to  Mr.  Horan,  for 
without  Ed  Horan  and  "Ma  Bell",  I’d  still  have  two  pieces  of 
equipment  that  "didn’t  want  to  talk  to  each  other". 

I  also  owe  my  most  sincere  appreciation  to  two  fellow 
students  and  friends,  Captains  Peter  Van  Wirt  and  Robert 
Lashlee.  These  individuals  helped  make  the  AFIT  experience  a 
highlight  in  my  life  -  THANKS  GUYS. 
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Abstract 

The  threats  of  dangerous  environments  and  projected  cuts 
in  military  personnel,  combined  with  advances  in  robotics  and 
sensors,  have  caught  the  attention  of  the  United  States  Air 
Force.  The  Flight  Dynamics  Laboratory  at  Wright-Patterson 
Air  Force  Base  has  been  conducting  research  into  concepts  for 
performing  aircraft  ground  refueling  using  robotic  systems. 

The  main  sensor  for  a  robotic  refueler  application  is 
vision.  Visual  information  received  from  a  TV  camera  mounted 
to  the  refueling  boom,  provides  the  feedback  data  necessary 
for  employing  visual  servo  control  techniques.  This  feedback 
data,  the  refueling  port’s  centroid  and  depth,  is  used  to 
visually  guide  the  robot  refueler  to  the  refueling  port. 

This  research  effort  designs  and  integrates  a  visual 
servo  control  scheme  for  a  PUMA  560  robot  arm  that  derives 
its  control  information  from  a  Machine  Intelligence 
Corporation  vision  system. —^The  vision  system’s  TV  camera  is 
mounted  to  the  PUMA’s  thirdyoint.  The  integrated  vision- 
robot  system  (VRS)  uses  closecNdoop,  static  and  dynamic, 
visual  servo  control  techniques  t\  demonstrate  the 
feasibility  of  ground  refueling.  \ 

The  results  successfully  demonstrate  the  VRS ' s  ability 
to  search  for  the  receptacle,  and  use  visual  servo  control 
techniques  to  guide  the  VRS  to  it. 
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IMPLEMENTATION  OF  A  VISUAL  SERVOING  SYSTEM 


FOR  EVALUATION  OF  ROBOTIC  REFUELING  APPLICATIONS 


I .  Introduction 


General  Issue 

Advances  in  robotic  technology  have  helped  to 
revolutionize  industry  worldwide.  The  major  reason  for  this 
revolution  lies  in  a  robot’s  ability  to  repeat  a 
preprogrammed  sequence  of  tasks.  Robots  have  freed  human 
beings  from  boring,  repetitive  tasks.  Their  strength  and 
endurance  allows  safe,  accurate  operations,  especially  in 
hazardous  conditions  which  exist  in  various  industrial  and 
military  situations.  A  robot  required  to  operate  effectively 
in  changing,  hazardous  environments  must  be  equipped  with 
sensors  (vision,  acoustic,  etc.)  and  artificial  intelligence 
(AT).  A I  algorithms  process  data  received  from  the  sensors 
enabling  real-time  decisions  to  be  made  affecting  the 
preprogrammed  operation  of  the  robot.  A  robot  equipped  with 
sensors  and  AI  is  considered  an  intelligent 
robot  (8:229-230). 

One  potentially  dangerous  environment  for  personnel 
exists  in  a  chemical/biological  (CB)  contaminated  area.  A  CB 
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contaminated  environment  on  the  runway  of  a  military 


installation  is  possible  during  a  hostile  situation. 

Exposure  to  this  environment  can  result  in  serious  injury  or 
death  ( 3 ) . 

One  solution  to  the  problem  of  exposing  personnel  to  a 
CB  environment  is  to  have  an  intelligent  robotic  refueler 
accomplish  the  task  of  aircraft  ground  refueling.  A  brief, 
simplified  scenario  of  a  possible  refueling  operation 
f  ol lows : 

1)  The  aircraft  parks. 

2)  The  robotic  refueler  pulls  up  to  the 
aircraft.  This  could  be  accomplished 
autonomously  or  by  using  a  human  driver. 

3)  A  refueling  boom  swings  up  and  over  the 
aircraft  near  the  air  refueling  port. 

4)  A  vision  system  finds  the  refueling  port  and 
vision  servo  control  techniques  guide  the 
robot  boom  into  the  port. 

5)  Once  the  aircraft  has  been  refueled,  the  boom 
is  removed  from  the  port  and  stowed. 

6)  The  robotic  refueler  pulls  away  from  the 
ai rcraft . 

7)  The  aircraft  departs. 

An  artist’s  conception  of  a  robotic  refueler  with  a  human 
driver  refueling  an  aircraft  through  the  air  refueling  port 
is  shown  in  Figure  1. 
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Background 

The  threats  of  dangerous  environments  and  projected  cuts 
in  military  personnel  have  raised  concern  in  several 
operational  aspects  of  the  United  States  Air  Force  (USAF). 

The  Flight  Dynamics  Laboratory  at  Wright  Patterson  Air  Force 
Base  has  been  conducting  research  into  concepts  for 
performing  aircraft  ground  refueling  using  robotic  systems. 

To  enable  a  robotic  refueler  to  operate  autonomously, 
intelligence  must  be  integrated  into  the  system.  The  main 
sensor  for  the  application  of  a  robot  refueler  is  vi s i on . 
Visual  information  received  from  a  television  camera  provides 
the  data  necessary  for  vision  servo  control.  Vision  servo 
control  provides  feedback  information  on  the  relative 
position  of  the  robot  to  a  target.  This  feedback  data  is 
used  to  guide  the  robot  to  the  desired  target  (12:107-116). 

Safety  requirements  in  current  ground  refueling 
procedures  require  a  minimum  of  three  military  members  to 
refuel  an  aircraft.  With  predicted  cuts  in  military 
manpower,  any  safe  refueling  method  which  could  reduce 
personnel  helps  solve  the  manning  problem.  It  is  also 
desirable  to  have  a  minimum  number  of  human  beings  exposed  in 
a  hostile  environment  during  the  refueling. 
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The  Flight  Dynamics  Laboratory’s  initial  research 
determined  the  air  refueling  port  to  be  the  best  place  to 
refuel  an  aircraft  using  a  robotic  refueler.  The  air 
refueling  port  has  many  advantages: 

1)  It  is  standard  on  all  Air  Force  fighter 
aircraft . 

2)  It  is  compliant  within  a  30  degree  cone  of 
insertion . 

3)  It  is  a  slipway  for  teleoperated  insertion  of 
a  boom  nozzle. 

4)  It  has  connections  for  audio  or  other  data 
transfer  information. 

Using  the  air  refueling  port  eliminates  the  need  for  robot 
end-effectors  to  simulate  complicated  human  dexterous 
manipulations  required  by  current  ground  refueling 
procedures,  such  as  opening  and  closing  an  access  panel, 
removing  and  replacing  the  fuel  cap,  and  twisting  on  and  off 
the  fuel  nozzle  (3). 

Statement  of  Problem 

The  purpose  of  this  research  is  to  design  and  integrate 
a  visual  servo  control  scheme  for  a  PUMA  560  robot  arm  that 
derives  its  control  information  from  a  Machine  Intelligence 
Corporation  (MIC)  vision  system.  The  aim  of  this  vision- 
robot  system  (VRS)  is  to  demonstrate  the  ability  of  a  ground 
refueling  robot  to  acquire  the  refueling  port  of  an  aircraft 
in  real-time.  Guidance  of  the  refueling  boom  into  the  port 
is  based  on  visual  feedback  received  from  a  television  camera 
rigidly  attached  to  the  robot  arm. 


Scope 

This  research  concentrates  on  visual  servoing.  The  MIC 
vision  system  takes  a  picture  of  a  well  defined  scene  and 
forms  a  video  grey-scale  binary  image  which  is  then  processed 
to  determine  the  position  and  orientation  of  a  target  in  the 
scene.  Binary  images  are  needed  for  the  fast  computer 
computations  required  during  efficient,  real-time  visual 
servo  control. 

This  investigation  focuses  on  three  separate  areas  to 
solve  the  problem  of  ground  refueling  an  aircraft  with  a 
visually  equipped  robot: 

1)  Teach  the  vision  system  to  recognize  the 
refueling  port  from  any  viewing  angle. 

2)  Design  a  visual  servo  controller  for  the 
real-time  guidance  of  the  robot  refueler  to 
the  air  refueling  port. 

3)  Develop  a  scanning  algorithm  that  moves  the 
robot  arm  in  a  pre-programmed  pattern  so 
its  vision  system  can  acquire  the  refueling 
port . 

Assumptions 

Since  the  cost  of  developing  a  full-scale  robot  refueler 
is  not  practical  for  this  project,  a  simple  simulation  of  the 
robot  refueler  will  demonstrate  the  theory  developed  during 
this  research.  To  simulate  video  acquisition  of  the 
refueling  port,  an  artificial,  well-defined,  high  contrast 
target-background  scene  was  constructed;  the  target,  a  white 
ball,  represents  the  refueling  port  and  a  black  background 
represents  the  surrounding  area.  The  vision-robot  system 
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(VRS)  will  scan  an  area  until  the  vision  system  acquires  the 
target.  Once  located,  the  visual  servo  controller  will  guide 
the  VRS  to  the  target. 

Computer  processing  time  is  of  the  utmost  importance  for 
effective,  real-time  visual  servo  control.  Determining  the 
location  and  orientation  of  a  ball  requires  the  least  amount 
of  computer  processing  in  the  MIC  image  processor,  because 
the  two  dimensional  image  of  a  ball  is  a  circle  from  any 
viewing  angle.  The  high  scene  contrast  assists  in  providing 
the  fast  processing  time  required  during  real-time  control 
operation.  White  against  black  also  eliminates  the  chance  of 
shadows  distorting  the  shape  of  the  ball.  A  lack  of  shadows 
removes  the  requirement  for  any  special  lighting  normally 
found  in  a  grey-scale  image  system. 

Another  reason  for  using  a  ball  as  the  target  is  the 
camera’s  fixed  focus  lens.  Even  though  the  ball  becomes 
blurry  as  the  camera  approaches  the  target,  the  ball's 
digital  image  will  remain  circular,  thus  allowing  simple 
target  identification. 

Additionally,  it  is  assumed  that  the  path  between  the 
ball  and  the  VRS  has  no  obstacles.  Therefore,  obstacle 
avoidance  techniques  will  not  be  researched  or  employed. 

Finally,  it  is  noted  that  an  added  capability  of  the  MIC 
vision  system  enables  it  to  determine  the  centroid  and 
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orientation  of  any  target  placed  in  its  camera's  field-of- 
view  (FOV).  This  enables  the  vision  system  to  provide  some 
information  about  a  target  even  though  it  may  not  be  able  to 
identify  the  target.  The  vision  system’s  television  camera 
will  be  mounted  rigidly  on  the  third  joint  of  the  PUMA  560 
robot  arm.  Specifications  concerning  the  PUMA  560  and  the 
MIC  vision  system  are  in  Appendix  A. 

General  Approach 

This  research  effort  will  be  separated  into  two  tasks 
based  on  two  different  types  of  visual  servo  control  schemes. 
The  first  task  implements  closed-loop,  static  look-and-move 
visual  servo  control  techniques  to  simulate  the  acquisition 
and  guidance  of  the  robot  refueler  boom  into  the  refueling 
port.  This  task  verifies  the  interface  and  correct  operation 
of  the  PUMA  560  and  the  MIC  vision  system.  It  also  develops 
a  system  that  searches  for  a  target,  and  once  acquired, 
implements  static  visual  servo  control  techniques  to  guide 
the  VRS  towards  the  target.  The  second  task  implements 
closed-loop,  dynamic  look-and-move  visual  servo  control 
techniques  to  simulate  the  guidance  of  the  robot  refueler 
boom  into  the  refueling  port. 
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Presentation 


This  thesis  is  composed  of  five  chapters.  Chapter  II 
summarizes  current  research  in  the  field  of  visual  servo 
control.  Chapter  III  presents  the  system  setup  and  research 
methodology  used  in  completing  this  research  effort.  Chapter 
IV  discusses  the  results.  Finally,  Chapter  V  presents  the 
conclusions  and  recommendations  for  further  work. 
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II.  Review  of  the  Literature 


Introduction 

The  use  of  robots,  especially  in  industrial  assembly, 
has  grown  at  a  remarkable  rate  during  the  last  two  decades. 

The  term  "robot"  was  coined  by  the  Czech  writer  Karel  Capek 
and  means  forced  labor.  Robots  do  the  work  of  humans 
although  they  do  not  look  or  act  like  humans  (8:2-3).  The 
major  advantage  of  robots  lies  in  their  ability  to  repeat  a 
preprogrammed  sequence  of  tasks  in  a  constant  environment 
with  great  accuracy.  The  major  disadvantage  of  robots  lies 
in  their  inability  to  sense  and  respond  to  a  changing 
environment  (8:229). 

The  integration  of  intelligence  into  a  robot  is  needed 
for  it  to  effectively  operate  in  a  changing  environment. 
Intelligence  can  be  installed  in  a  robot  by  equipping  it  with 
appropriate  sensors  and  artificial  intelligence  (AI).  Figure 
2  shows  a  block  diagram  of  a  possible  intelligent  robot.  Data 
from  the  sensors  provide  the  necessary  information  required 
by  AI  algorithms  to  enable  real-time  decisions  that  may 
affect  the  preprogrammed  operation  of  the  robot  (8:229-230). 

This  chapter  briefly  introduces  the  various  robotic 
sensors  and  control  schemes  currently  in  use  or  in 
development.  A  summary  of  the  various  applications  of  robot 
manipulator  systems  integrated  with  visual  sensors  and  visual 
servo  control  schemes  will  also  be  presented. 
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Figure  2.  Block  Diagram  of  an  Intelligent  Robot  System 

(8:230) 


Sensors 

Sensor  development  and  integration  has  been  an  area  of 
active  research.  "The  interaction  of  robots  with  sensors  has 
always  been  an  important  goal  in  the  development  of  robotic 
systems.  Such  sensor-based  systems  would  have  increased 
functional  capabilities  as  well  as  flexibility  in  the 
execution  of  tasks"  (12:107).  Robotic  sensors  are  classified 
as  either  contact  or  noncontact.  Contact  sensors  include 
tactile  and  force-torque  sensors.  Noncontact  sensors 


include  proximity,  acoustic,  range,  and  visual  sensors  (8:230). 


The  visual  sensor  of  interest  to  this  research  uses  an 


image  processor  to  digitize  a  scene  viewed  through  a 
television  (TV)  camera.  The  vision  system  provides  the 
position  and  orientation  of  a  target.  This  information  is 
fed  back  to  the  robot  controller.  The  controller  directs  the 
robot  to  the  target,  enabling  the  robot  to  complete  its 
programmed  tasks. 

Control 

The  two  basic  types  of  control  schemes  employed,  when 
using  sensor  data  for  robot  control,  are  static  sensing  and 
dynamic  sensing.  The  static  sensing  method  maintains  the 
robot  in  a  stationary  position  while  sensing  takes  place. 

Most  control  systems  which  implement  this  method  use  open 
loop  control.  Once  all  processing  of  sensor  data  has  been 
completed,  the  robot  accomplishes  its  required  tasks  without 
any  further  information  received  from  the  sensing  device.  By 
contrast,  the  dynamic  method  controls  real-time  robot  motion 
based  on  informational  updates  acquired  from  the  sensor  in  a 
closed  loop  configuration  (8:231). 

Most  modern  vision  systems  employ  a  static  closed-loop 
mode  of  operation.  The  vision  system  takes  a  target’s 
picture  and  feedbacks  its  position  with  respect  to  the 
robot’s  position.  The  difference  between' the  two  positions 
is  considered  a  sensed  error.  The  sensed  error  actuates  the 
necessary  robot  movements  required  to  eliminate  the  error. 

The  vision  system  continually  monitors  the  error  between  the 


robot’s  position  relative  to  the  target’s  after  each  robot 
motion  until  the  error  no  longer  exists  (8:231). 

Visual  Servo  Control 

A  limited  amount  of  research  has  been  done  in  the 
development  of  control  structures  for  visual  servoing.  This 
section  discusses  current  techniques  and  approaches  used  in 
visual  servo  control. 

"The  function  of  machine  vision  in  servoing  is  to 

determine  the  spatial  relationship  that  exists  between  the 

camera,  tool,  and  the  workpiece"  (1:943). 

Visual  servo  robot  control  systems  provide  feedback 
on  the  relative  end-effector  position  of  a  robot. 

They  offer  an  interactive  positioning  mechanism 
which  depends  upon  extraction  and  interpretation  of 
visual  information  from  the  environment  (12:107). 

The  accuracy  of  visual  feedback  depends  on  the  distance  from 

the  camera  to  the  end-effector  and  the  distortion  of  the 

image  plane  due  to  the  camera  lens  (1:944-945). 

Visual  servo  control  systems  are  characterized  by  either 

a  feedback  representation  or  a  joint  control  mode .  Image- 

based  and  position-based  visual  feedback  systems  comprise  the 

feedback  representation.  Image-based  visual  feedback  control 

parameters  are  based  on  an  image’s  features,  where  as 

position-based  visual  feedback  control  parameters  are  based 

on  an  object’s  geometry,  position  and  orientation  (12:108). 

The  two  basic  joint  control  modes  are  closed-loop  and 

open-loop.  "The  ’ look-and-move ’  structures  utilize  inner 

closed-loop  joint  control.  The  ’visual-tracking’  structures 
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have  no  closed-loop  joint  control  and  rely  only  on  sensory 

feedback  to  drive  the  manipulator"  (12:108). 

Practical,  real-time  operation  of  sensor  servo  systems 

has  yet  to  be  achieved.  Sanderson  and  Weiss  believe: 

A  number  of  factors  have  delayed  the  practical 
development  of  sensor-based  servo  systems.  Robot 
positioning  systems  and  robot  control  systems  are 
difficult  to  analyze  and  design  in  themselves,  and 
there  are  a  large  number  of  practical  applications 
of  non-sensing  robots  in  highly  structured 
industrial  and  other  environments.  Sensing 
systems,  particularly  vision,  are  often  slow 
relative  to  manipulator  dynamics,  and  practical 
applications  or  machine  vision  are  currently  also 
limited  to  highly  constrained  situations.  The 
analytical  complexity  of  both  manipulator  control 
and  sensory  data  interpretation  make  general 
formulation  of  the  sensor-based  control  problem 
challenging  (12:107). 

In  visual  control  systems,  the  video  camera  provides  the 
required  image  information  for  determining  camera  position 
relative  to  object  position.  Computer  processing  times 
(including  interprocessor  communications)  under  1/10  of  a 
second  (14:214)  are  required  during  scene  analysis  to  achieve 
real-time  visual  servoing  for  some  typical  engineering 
applications.  The  computer  processing  includes  image 
preprocessing,  feature  extraction,  and  interpretation.  The 
computer  processing  adds  unwanted  noise  and  time  delays  to 
the  system  which,  when  taken  into  account,  adds  complexity  to 
the  system’s  operation. 

A  majority  of  the  control  work  using  vision  systems  has 
been  with  the  open  loop,  static  1 ook-and-move  type  discussed 
earlier.  In  these  applications  the  camera  is  mounted 
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separately  from  the  robot.  The  vision  system  takes  a 
picture,  processes  the  scene  information,  and  outputs  the 
object’s  position  and  orientation  to  the  robot  control  system 
so  the  robot  manipulator  can  accomplish  its  predetermined 
task.  This  technique  suffers  since  the  above  operations  do 
not  occur  in  the  real-time  requirement  for  visual  servo 
control . 

Research 

Numerous  authors  have  researched  and/or  developed  a 
variety  of  vision-robot  systems  using  some  form  of  visual 
servoing.  There  are  two  basic  methods  used  in  vision 
servoing.  One  involved  the  camera  mounted  over  the  workspace 
while  the  other  has  the  camera  mounted  directly  on  the  robot 
manipulator.  The  first  method  usually  works  on  systems 
concerned  with  the  servoing  of  a  robot  to  grasp  an 
unorientated  object.  Ward  et  al . ,  developed  CONSIGHT:  an 
industrial  vision-based  robot  system  which  uses  a  linear 
array  camera  (mounted  overhead)  and  structured  light.  As  an 
object  passes  below  the  camera  on  a  moving  belt,  CONSIGHT 
produces  a  two-dimensional  ( 2  —  D )  image  of  the  object  in  order 
to  determine  its  position  and  orientation  with  respect  to  the 
robot  (15:195-211).  This  system,  as  well  as  others  using 
structured  light,  only  work  effectively  in  controlled, 
relatively  stable  environments  like  those  found  in  many 
industrial  applications. 
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Another  example  using  a  camera  mounted  separately  from 
the  robot  arm  was  done  by  Palm  et  al .  They  have  demonstrated 
how  a  visual  system  comprised  of  a  single  camera  and  two 
light  sources  used  visual  servoing  techniques  to  enable  the 
accurate  robotic  assembly  of  pins  into  an  electrical 
connector.  The  vision  algorithm  relied  upon  the  relative 
distance  between  the  pin's  actual  position  and  its  desired 
position  (10:221-236). 

The  second  method  used  in  visual  servoing  has  the  camera 
mounted  on  the  robot  manipulator.  "This  technique  is 
desirable  for  accuracy  due  to  the  close  proximity  of  the 
camera  to  the  end-effector  and  the  workpiece"  (1:943). 
Research  in  this  area  involves  the  extraction  of  3-D 
information  from  a  2-D  image  using  triangulation,  structured 
lighting  ,  and  perspective  cue  depth  measurement  (1:943). 

This  method  has  been  proven  successful  by  Hill  and  Park,  and 
VanderBrug  (6;  14).  VanderBrug  et  al .  developed  a  system 
utilizing  a  miniature  solid  state  camera  and  a  strobographic 
light  source  mounted  directly  onto  the  robot  manipulator. 

The  light  source  flashes  a  plane  of  light  onto  an  object  in 
the  camera’s  field  of  view.  When  the  plane  of  light  strikes 
the  object,  line  segment  images  are  viewed  by  the  camera. 
Computer  algorithms  utilizing  triangulation  principles 
interpret  the  image  segments  to  provide  the  range  and 
orientation  of  the  object  (14:213-231). 
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Bamba  et  al .  incorporated  a  system  based  on  the  optical 
pattern  projection  method  used  for  recognition  of  3-D 
objects.  Their  system  uses  a  small  visual  sensor  which  has  a 
photo  sensitive  position  detector  and  a  light  emitting  diode 
(LED)  for  its  main  components.  This  system  can  be  used  for 
path  correction  during  arc-welding.  Light  emitted  from  the 
LED  is  projected  onto  the  surface  of  the  object  to  be  welded 
and  deflected  back  to  the  position  detector,  providing  3-D 
information  about  the  object  (2:169-177). 

The  systems  mentioned  above  all  suffer  from  the  same 
problem;  the  requirement  for  some  form  of  structured  light 
and  controlled  environments.  These  systems  would  not  be 
practical  for  a  robotic  ground  refueler  which  would  have  to 
operate  in  a  changing  environment.  The  next  three 
developments  more  closing  relate  to  the  scope  of  this 
research  effort. 

An  example  using  a  small,  solid  state  camera  attached  to 
a  robot  end-effector  was  demonstrated  by  Hill  and  Park.  This 
system  can  operate  by  either  conventional  lighting  or 
structured  lighting,  therefore,  making  it  more  versatile. 

The  image  processor  converts  a  scene  into  binary  images  to 
achieve  fast  and  reliable  processing.  Visual  feedback 
enables  the  robot  to  guide  the  manipulator  to  a  desired 
target  (6:233-246). 

Harrell  et  al.,  developed  a  visual  servo  system  for  a 
three  degree-of-f reedom  harvest  robot  to  pick  fruit  from  a 
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tree.  The  system  received  its  visual  feedback  information 
from  a  CCD  camera  mounted  in  the  robot’s  third  prismatic 
joint.  "Simple  proportional  gain  control  laws  utilized  this 
vision  feedback  to  independently  servo  the  two  revolute 
joints"  (5:537)  to  the  desired  piece  of  fruit.  The  system 
used  a  strobe  light  to  determine  the  distance  to  the  fruit. 
The  system  successfully  tracked  fruit  motions  during  a 
harvest  cycle  (5:537-545). 

A  system  that  did  not  need  to  compute  the  coordinate 
relationship  between  the  robot  end-effector  and  the  camera 
was  developed  by  Kim  et  al .  The  system  used  a  miniature 
camera  mounted  to  the  sixth  joint  of  the  PUMA  560  robot  arm. 
The  system  visually  servoed  itself  to  the  correct  orientation 
and  position  to  grasp  a  target.  The  system  obtained  3-D 
information  about  the  target  from  a  single  camera  which  used 
an  iterative  focusing  method  and  a  distance  formula  which 
derived  distance  from  two  different  images  along  the  same 
line  of  sight  of  the  camera  (7:417-422). 

Summary 

"Visual  servo  robot  control  systems  provide  feedback  on 
the  relative  end-effector  position  of  a  robot.  They  offer  an 
interactive  positioning  mechanism  which  depends  upon 
extraction  and  interpretation  of  visual  information  from  the 
environment"  (12:115).  Most  current  systems  operate  using 
the  basic  static  look-and-move  sequence  of  operations.  These 
systems  usually  incorporate  a  camera  mounted  over  the 
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workspace.  However,  there  is  increasing  development  in  the 
areas  of  integrated  vision-robot  systems  using  dynamic  look- 
and-move  and  visual-tracking  techniques.  These  new 
developments  usually  have  the  camera  mounted  directly  on  the 
robot  manipulator.  An  intelligent  robot  is  "visionable"  by 
equipping  it  with  the  ability  to  sense  and  respond  to  a 
changing  environment. 

This  completes  the  Review  of  the  Literature;  the  system 
setup  and  methodology  can  now  be  presented. 
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Introduction 


III.  System  Setup  and  Methodology 


The  goal  of  this  research  is  to  design  and  implement  a 
visual  servo  control  scheme  to  simulate  the  ground  refueling 
of  an  aircraft  using  a  visually  guided  robot.  An  integrated 
vision-robot  system  (VRS)  composed  of  a  PUMA  560  robot  arm 
and  MIC  vision  system  are  used  to  accomplish  the  research. 

This  chapter  describes  the  system  setup  and  methodology 
involved  in  applying  static  and  dynamic,  look-and-move  visual 
servo  control  schemes  to  an  integrated  robot-vision  system. 

Static  Look-And-Move 

To  accomplish  the  task  of  developing  a  system  that 
searches  for  and  acquires  a  target  using  static  look-and-move 
visual  servo  control,  the  task  was  separated  into  the 
following  four  subtasks: 

1.  Develop  an  open  loop,  static  look-and- 
move  system  with  the  camera  mounted 
above  the  robot  work  space. 

2.  Develop  a  closed  loop,  static  look-and-move  visual 
servo  control  system  with  the  camera  mounted  to  the 
third  joint  of  the  PUMA  560  that  tracks  a  target 
(white  ball)  against  a  black  background. 

3.  Using  the  results  from  Subtask  2,  develop  a  visual 
servo  control  system  that  tracks  and  moves  towards 
the  target. 

4.  Add  to  Subtask  3  by  developing  a  scanning  algorithm 
to  search  for  and  acquire  the  target. 

The  accomplishment  of  each  subtask  serves  as  a  building  block 
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in  achieving  the  overall  task.  The  following  subsections 
describe  each  subtask’s  system  setup  and  methodology. 

Subtask  1 ;  Camera  Mounted  Above  the  Work  Space . 

This  subtask  verifies  the  interface  between  the  MIC 
system  and  the  PUMA  560  robot  arm.  It  introduces  the 
following:  the  PUMA  560  robot  arm  and  its  associated 
peripherals;  the  VAL  II  programming  language;  the  MIC  vision 
system;  the  interface  between  the  vision  system  and  the  PUMA 
560;  and  the  relationship  between  locations  in  the  camera’s 
frame  and  the  robot’s  frame.  A  description  of  the  PUMA  560, 
the  MIC  vision  system,  and  the  interface  between  the  two 
systems  is  in  Appendix  A. 

The  camera  is  mounted  to  a  camera  stand  and  positioned 
above  a  table  in  the  robot’s  work  space  as  shown  in  Figure  3. 
The  vision  system’s  effective  work  area  is  established  by  the 
camera’s  f ield-of -view  (FOV).  The  camera’s  FOV  is  determined 
by  the  type  of  lens  used  and  its  focal  length,  and  is 
described  by  the  height  and  width  of  the  visual  field.  The 
lens  used  in  this  research  is  a  Fujinon  TV  1:1.4/25  and  has  a 
FOV  with  a  height  of  300  mm  and  width  of  365  mm. 

The  image  processor  in  the  vision  system  transforms  the 
grey-scale  camera  image  into  a  binary  image  by  means  of  a 
programmable  threshold  in  order  to  generate  its  silhouette. 

To  simplify  the  threshold  operation,  images  should  be 
selected  with  maximum  contrast  between  the  target  and  its 
background.  White  targets  against  a  black  background  are 
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used  in  this  subtask.  A  white  target  against  a  black 


background  also  eliminates  shadows,  thereby  abolishing  any 
requirement  for  special  lighting. 


Figure  3.  Camera  Mounted  Above  the  Work  Space 

An  open  loop,  static  look-and-move  system  is  implemented 
to  verify  the  interface  between  the  vision  system  and  the 
robot.  The  vision  system  identifies  a  target  and  provides 
the  two-dimensional  location  of  its  centroid.  This 
information  is  transmitted,  open  loop,  to  the  robot 
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controller  enabling  the  robot  to  move  to  the  desired 
location.  A  functional  block  diagram  of  the  open  loop 
vision-robot  control  system  is  shown  in  Figure  4. 
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A  menu  driven  software  package  accomplishes  the  tasks 
shown  in  Figure  5.  Appendix  B  lists  the  various  programs 
used  by  this  package  (NOTE:  listings  of  the  system  programs 
are  proprietary  information  and  not  provided.  However, 
brief  descriptions  of  the  software  are  in  Appendix  A). 
Appendix  C  contains  a  user’s  manual.  The  ’WARNING’  is 
required  to  prevent  unpredicted  movements  of  the  arm  due  to 
inaccurate  calibration.  Figure  6  shows  the  software 
structure  of  the  various  user  selectable  options.  Option  1 
initializes  communication  between  the  vision  system  and  the 
robot . 
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THE  FOLLOWING  OPTIONS  ARE  AVAILABLE 

1.  INITIALIZE  VISION/ROBOT  COMMUNICATION 

2.  VISION-TO-ROBOT  CALIBRATION 

3.  PROTOTYPE  TRAINING 

4.  PROTOTYPE  STORAGE  AND  DELETION 

5.  VISION  DEMO 

********************************************* 

- >  WARNING  < - 

BEFORB  EXECUTING  OPTIONS  3,4,  OR  5, 
OPTIONS  1  AND  2  MUST  BE  ACCOMPLISHED 

********************************************* 

PLEASE  BNTER  THE  NUMBER  OF  THE  OPTION 
YOU  DESIRE  - > 
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Figure  5.  Vision  Menu 
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Figure  6 .  Software  Structure  Chart 


Option  2  calibrates  the  vision-robot  system.  This 
option  determines  the  scale  factor  defining  the  ratio  of  the 
distance  between  two  locations  in  the  robot’s  frame  to  the 
distance  in  the  camera’s  frame.  Finally,  the  vision-robot 
work  frame  is  established  to  determine  the  relative 
transformation  between  the  camera  and  the  robot. 

Options  3  and  4  deal  with  prototyping.  Option  3  allows 
the  user  to  teach  the  vision  system  various  targets,  such  as 
a  circle  or  rectangle.  Option  4  enables  the  user  to  store, 
recall,  or  delete  the  various  targets  trained. 
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Option  5  calls  another  menu  which  runs  a  demo  permitting 


the  user  to  run  various  programs  which  identify  trained 


targets  and/or  directs  the  robot  to  point  to  the  centroid  of 


a  target  in  the  camera’s  FOV.  Figure  7  displays  the  various 


user  selectable  options  in  the  demo.  Appendix  B  lists  the 


programs  used  in  the  demo  options.  Figures  8  displays  the 


software  structure  chart.  Demo  1  instructs  the  vision  system 


to  take  a  picture  to  determine  the  number  of  targets,  and  the 


identification  and  centroid  position  of  the  largest  target  in 


the  camera’s  FOV.  Demos  2  and  3  accomplish  the  same  tasks  as 


Demo  1  with  the  addition  of  instructing  the  robot  to  point  to 


the  desired  target’s  centroid. 
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Figure  7.  Vision  Demo  Menu 
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Figure  8.  Software  Structure  Chart 


With  the  robot  and  vision  systems  introduced  and  the 
interface  verified,  the  remaining  three  subtasks  will 
describe  the  setup  and  methodology  for  developing  a  visually- 
guided  robot  refueler. 

Subtask  2 :  Camera  Mounted  on  Arm ;  Tracking 

This  subtask  mounts  the  camera  onto  the  third  joint  of 
the  PUMA  560  and  implements  a  closed  loop,  static  look-and- 
move  visual  servo  control  scheme.  The  control  scheme 
utilizes  information  from  the  vision  system  to  provide  the 
location  of  a  target’s  centroid  to  the  robot  controller.  The 
controller  uses  the  location  information  to  control  the 
motion  of  the  robot  in  the  PUMA’s  world  coordinate  frame. 

The  PUMA  560  manipulator  is  controlled  point-to-point. 


Motion  between  points  can  be  controlled  by  a  proportional  (P) 
or  proportional  and  derivative  (PD)  feedback  loop.  The  P 


feedback  loop  uses  high  gain  position  feedback  to  bring  the 
difference  between  the  robot’s  desired  position  and  the 
actual  position  to  zero.  However,  a  P  feedback  loop 
controller  cannot  guarantee  an  overdamped  response,  i.e.  no 
overshoot  (4:89-92).  This  visual  servo  control  scheme  cannot 
afford  a  problem  with  overshoot,  because  the  VRS  may  strike 
the  target  or  oscillate  around  it  endlessly.  To  overcome 
this  problem,  the  PD  feedback  loop  controller,  which  feeds 
back  velocity  as  well  as  position  is  used.  This  system  is 
tuned  by  the  manufacturer  to  provide  an  overdamped  response; 
guaranteeing  no  overshoot.  PD  loop  control  is  desired  for 
this  research  effort,  however,  the  PD  loop  is  only  active  for 
28  millisecond  increments.  Therefore,  robot  movement 
commands  must  be  divided  into  motions  that  can  be  completed 
during  each  time  increment,  otherwise  P  loop  control  takes 
over  and  an  overdamped  response  is  not  guaranteed. 

It  is  desired  to  have  the  camera  centered  on  the  target 
at  all  times.  Therefore,  as  the  target  moves,  the  robot  will 
track  it,  attempting  to  keep  the  target  centered  in  the 
camera’s  FOV .  The  vision  system  takes  a  picture  and 
estimates  the  error  between  the  target’s  position  and  the 
center  of  the  camera’s  FOV.  The  block  diagram  of  the  visual 
servo  control  scheme  is  shown  in  Figure  9.,  This  subtask 
simulates  the  visually  guided  robot  refueler’s  responsibility 
to  maintain  camera  view  on  the  aircraft  refueling  port. 
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Figure  9.  Block  Diagram  of  Visual  Servo  Control  Scheme 


The  camera  is  mounted  to  the  robot’s  third  joint  using 
Velcro  strips.  The  camera  lens  is  pointed  perpendicularly  to 
the  robot’s  World  Coordinate  System  (WCS)  x-y  plane,  see 
Figure  10.  In  this  configuration,  an  offset  exists  between 
the  location  of  the  camera  and  the  robot’s  end-effector.  To 
account  for  the  offset,  the  center  of  the  camera’s  FOV  is 
considered  to  be  the  robot  end-effector’s  current  location. 
This  is  a  translation  along  the  camera’s  viewing  axis.  An  x 
and  y  correction  is  not  necessary  since  it  is  desired  to  keep 
the  target  centered  in  the  camera's  FOV,  not  centered  below 
the  robot  end-effector.  Therefore,  if  the  target  is  not 
centered  in  the  camera’s  FOV  '/hen  a  picture  is  taken,  the 
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The  vision-robot  system  is  calibrated  to  relate  camera 
pixel  positions  in  the  camera’s  x-y  plane  to  robot  positions 
in  the  robot’s  WCS  x-y  plane.  Since  this  work  is  two- 
dimensional  and  the  x-y  plane  of  the  camera’s  FOV  is  parallel 
to  the  robot's  x-y  plane,  complicated  transformations 
relating  the  position  of  a  target  in  the  camera’s  FOV  to  the 
location  of  the  robot's  end-effector  are  not  required.  Thus, 
during  Subtask  2,  the  system  is  not  interested  in  the 
coordinate  relationship  between  the  robot’s  location  with 
respect  to  a  target  placed  in  the  camera’s  FOV.  The  vision 
system  simply  provides  the  x  and  y  offset  position  of  the 
target  with  respect  to  the  robot's  current  x  and  y  position. 
Therefore,  a  simple  one-to-one  relationship  exists  between 
the  location  of  the  target  in  the  camera’s  FOV  and  the 
robot’s  work  space. 

The  computer  program,  track. targ,  (whose  flow  diagram  is 
shown  in  Figure  11  and  program  listing  is  in  Appendix  B)  uses 
information  from  the  vision  system  to  correct  any  offsets 
between  the  centroid  position  of  the  target  and  the  center  of 
the  camera’s  FOV.  Once  the  target  is  placed  anywhere  in  the 
camera’s  FOV,  its  centroid  position  is  transmitted  to  the 
robot  controller.  This  provides  the  information  necessary  to 
command  the  robot  to  position  the  arm,  such  that  the  center 
of  the  camera’s  FOV  is  directly  over  the  target’s  centroid. 
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Subtask  3 :  Camera  on  Arm;  Tracking  and  Acouirin; 


This  subtask  builds  on  the  second  subtask  by  instructing 
the  vision-robot  system  to  move  towards  the  target  a 
predetermined  distance  while  it  is  tracking.  This  simulates 
the  refueling  nozzle  approaching  the  refueling  port.  The  same 
closed  loop,  static  look-and-move  visual  control  scheme  is 
employed,  with  the  additional  capability  of  enabling  the  VRS 
to  move  towards  the  target. 

To  provide  the  VRS  with  the  distance  of  the  target  to 
the  camera,  the  vision  system  is  trained  with  the  target  at 
nine  different  locations,  25  mm  apart,  along  the  camera’s 
pointing  axis.  When  the  target  is  at  different  distances 
from  the  camera,  the  image  size  varies.  The  larger  the 
image,  the  closer  the  camera  is  to  the  target.  During 
training  of  the  vision  system,  the  target  is  placed  in  the 
first  position  at  the  maximum  distance  from  the  camera  lens. 
In  the  ninth  position,  the  target  is  trained  at  the  closest 
position  to  the  camera.  Thus,  the  larger  the  target  in  the 
camera's  FOV,  the  closer  the  target  is  to  the  vision-robot 
end-ef  f  ector . 

A  white  ball  is  used  as  the  target  to  alleviate  one 
problem  encountered  with  a  fixed  focus  camera  lens.  As  the 
camera  moves  closer  to  the  ball,  the  camera  image  becomes  out 
of  focus.  However,  the  image  still  remains  circular  in  the 
camera’s  FOV  and  the  binary  threshold  computation  produces 
sharp  boundaries  for  the  centroid  computation. 


The  computer  program,  track . targ . scat ,  (whose  partial 
flow  diagram  is  shown  in  Figure  12  and  program  listing  is  in 
Appendix  B)  instructs  the  vision  system  to  take  a  picture  and 
recognize  one  of  the  trained  targets.  Depending  on  which  of 
the  nine  targets  is  identified,  the  VRS  knows  the  height  of 
the  target  from  the  camera.  This  method  works  fine  for  a 
well  defined  target  like  a  ball,  however,  an  improved  system 
is  required  for  a  more  complicated  image. 

The  procedure  described  above,  enables  the  VRS  to  obtain 
three  dimensional  (x,  y,  z)  information  about  the  target 
placed  in  the  camera's  FOV.  The  vision  system  takes  a 
picture  to  determine  the  x  and  y  centroid  position  of  the 
target.  The  z  position  is  determined  once  one  of  the  nine 
target  images  has  been  identified. 

When  a  picture  is  taken,  the  VRS  approaches  the  target. 
To  account  for  positional  errors  inherent  in  the  vision-robot 
system  controller,  the  system  converges  towards  the  target  in 
increments,  allowing  the  system  to  keep  the  target  centered 
in  the  camera’s  FOV.  This  convergence  method  minimizes  the 
tracking  error.  The  target  identified  determines  the 
incremental  distance  the  system  will  move  towards  the  target. 
Another  picture  is  taken  and  the  process  is  repeated  until 
the  VRS  is  at  the  desired  distance  and  position  from  the 
target . 
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Figure  12.  Software  Flow  Diagram  of  track . targ . scat 
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Subtask  4 :  Camera  on  Arm :  Search 

This  subtask  completes  the  overall  task  of  designing  and 
implementing  a  vision-robot  system  that  searches  for  and 
acquires  a  ball. 

The  computer  program,  search,  (whose  partial  flow 
diagram  is  shown  in  Figure  13  and  program  listing  is  in 
Appendix  B)  instructs  the  VRS  to  go  through  a  square  conical 
search,  see  Figure  14,  covering  the  perimeter  of  the  search 
area  first,  then  moving  towards  the  center  of  the  area.  The 
VRS  moves  in  increments  approximately  equivalent  to  half  the 
camera’s  FOV .  Moving  in  this  fashion  provides  an  overlap  to 
ensure  the  search  covers  the  entire  search  area.  The  search 
is  broken  up  into  16  increments,  as  shown  in  Figure  14.  Each 
time  the  robot  moves  in  the  search  pattern,  a  picture  is 
taken  to  check  if  the  ball  is  in  the  camera's  FOV.  Once  the 
target  has  been  acquired  by  the  VRS,  the  procedures  from 
Subtask  3  are  executed  and  the  VRS  visually  servos  itself 
towards  the  target. 

This  completes  the  description  of  the  system  setup  and 
methodology  for  the  closed  loop,  static  look-and-move  visual 
servo  control  scheme.  The  dynamic  look-and-move  system  is 
presented  in  the  next  section. 
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Figure  13.  Software  Flow  Diagram  of  search 
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Dynamic  Look-And-Move  System 

In  this  task,  a  closed  loop,  dynamic  look-and-move 
visual  servo  control  system  is  designed  and  implemented. 

This  task  repeats  much  of  the  work  accomplished  in  Subtask  3. 
The  difference  being  in  the  approach- to- target  motion  of  the 
robot  arm.  The  controller  implemented  in  this  task  uses 
dynamic  visual  servoing  to  guide  the  arm  towards  the  ball. 
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A  system  with  parallel  processing  is  required  to 
achieve  true  closed-loop,  dynamic  visual  servo  control. 
Because  of  computer  speed  limitations,  this  capability  is 
currently  not  possible  for  this  integrated  vision-robot 
configuration.  However,  a  form  of  dynamic  control  using 
serial  program  execution  is  possible  with  the  present 
configuration.  In  VAL  II  program  execution,  once  a  command 
has  been  executed,  the  processor  immediately  executes  the 
next  instruction,  even  if  the  last  instruction  has  not  been 
completed.  Unimation  refers  to  the  procedure  as  "procedural 
motion"  (17).  Thus,  when  a  command  to  process  a  picture  is 
executed  before  arm  motion  has  stopped,  the  requirements  for 
dynamic,  visual  servo  control  are  met.  The  situation  exists 
when  the  arm  moves  towards  the  target. 

As  the  arm  moves  towards  the  ball,  the  computer  program, 
track . targ . dcat ,  (whose  flow  diagram  is  shown  in  Figure  15 
and  program  listing  is  in  Appendix  B)  instructs  the  vision 
system  to  take  a  picture  and  provide  real  time  corrections 
for  any  offsets  occurring  as  the  arm  moves  towards  the  ball. 

This  completes  the  description  of  the  system  setup  and 
methodology  for  this  thesis;  the  results  are  now  presented. 
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Figure  15.  Software  Flow  Diagram  of  track . targ . dcat 
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IV.  Results  and  Discussion 


Introduction 

The  goal  of  this  thesis  was  to  develop  an  integrated 
vision-robot  system  (VRS)  to  visually  acquire  a  target.  The 
VRS  implemented  two  visual  servo  control  techniques  to  guide 
the  system  to  the  target.  These  two  techniques  included 
static  and  dynamic,  look-and-move  visual  servo  control.  Each 
control  technique  was  considered  a  separate  task  during 
development  and  testing.  The  testing  of  each  task  of  the 
integrated  VRS  occurred  simultaneously  during  development. 

The  results  from  each  task  were  directly  applied  to 
subsequent  tasks.  This  chapter  describes  the  results  from 
each  task  in  the  following  sections. 

Static  Look-And-Move 

The  task  of  developing  a  static  look-and-move  visual 
servo  control  system  was  divided  into  the  following  four 
subtasks : 

1.  Develop  an  open  loop,  static  look-and-move 
system  with  the  camera  mounted  above  the  robot. 

2.  Develop  a  closed  loop,  static  look-and-move  visual 
servo  control  system  with  the  camera  mounted  to  the 
third  joint  of  the  PUMA  560  that  tracks  a  target 
(white  ball)  against  a  black  background. 

3.  From  Subtask  2,  develop  a  visual  servo  control 
system  to  track  and  approach  tne  target. 

4.  Add  to  Subtask  3  by  developing  a  scanning  algorithm 
to  search  for  and  acquire  the  target. 

These  subtasks  are  described  in  the  following  subsections. 
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Subtask  1 :  Camera  Mounted  Above  the  Work  Space . 

The  first  subtask  was  to  develop  an  open  loop,  static 
look-and-move  scheme  which  has  the  video  camera  mounted  above 
the  workspace.  The  purpose  was  to  introduce  the  equipment 
and  verify  the  software  and  hardware  interfaces  between  the 
vision  system  and  the  robot.  In  this  scheme,  the  camera  was 
mounted  to  a  stand  and  positioned  above  the  robot  workspace. 

The  hardware  and  software  interface  was  verified,  the 
vision-robot  system  calibrated,  and  the  vision-robot  work 
frame  established  by  following  the  procedures  in  the 
Univision  User’s  Manual.  In  addition,  two  targets  were 
trained  to  the  vision  system;  a  white  2  inch  diameter  circle 
and  a  white  3  by  5  inch  rectangle,  see  Figure  16. 


Figure  16.  Targets  Used  in  Subtask  1. 
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Once  the  targets  were  trained  to  the  vision  system,  the 


program  vision. demo,  see  Appendix  B,  was  executed.  The 
programs  executed  in  this  demonstration  successfully  achieved 
the  open  loop,  static  look-and-move  configuration.  Each  time 
a  program  was  executed,  the  vision  system  identified  the 
target  and  transmitted  its  orientation  and  two-dimensional 
centroid  position  to  the  robot  controller. 

The  first  program  in  the  demonstration,  ident. blobs,  see 
Appendix  B,  determined  the  number  of  targets  in  the  camera’s 
FOV,  identified  the  largest  target,  and  provided  the 
orientation  and  two-dimensional  centroid  position  of  the 
largest  target  in  the  FOV.  This  program  also  instructed  the 
vision  system  to  highlight  each  target  on  the  video  monitor 
with  a  white  border.  It  was  observed,  for  the  processing  of 
the  rectangular  target,  that  the  inter-processor 
communication  and  image  processing  took  approximately  2.3 
seconds  during  program  execution.  This  time  varies  depending 
on  how  many  targets  are  processed.  Figures  17,  18,  and  19 
show  the  rectangular  target  placed  in  the  work  frame,  the 
highlighted  digitized  image  displayed  on  the  vision  system's 
video  monitor,  and  the  output  to  the  user’s  terminal. 
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Figure  19.  User  Terminal  Output 


The  second  program,  ident . blob . point . center ,  see 
Appendix  B,  accomplished  the  same  tasks  as  ident. blobs,  with 
the  addition  of  instructing  the  robot  controller  to  position 
the  robot  end-effector  above  the  center  of  the  target. 

Figures  20,  21,  and  22  show  the  circular  target  placed  in  the 

work  frame,  the  digitized  image  displayed  on  the  vision 
system’s  monitor  once  the  image  was  processed,  and  the  final 
position  of  the  robot’s  end-effector  after  program  execution. 
Figure  23  shows  the  results  displayed  on  the  robot  terminal. 
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Figure  22.  Final  Position  of  Robot  End-Effector 
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Figure  23.  User  Terminal  Output 


The  third  program,  point . center ,  see  Appendix  B, 
accomplished  the  same  tasks  as  ident . blob . point . center , 
except  it  did  not  highlight  or  identify  the  targets.  This 
minimized  image  processing  thereby  allowing  faster  operation. 

During  th u  testing  of  the  second  and  third  programs, 
several  obse- vations  indicated  parallax  induced  errors  in  the 
end-effector’s  final  position  over  the  target  when  the  target 
was  placed  towards  the  edges  of  the  work  frame  (see  Figure 
24).  However,  this  thesis  was  focused  on  visual  servo 
control,  therefore,  these  errors  were  judged  as  insignificant 
and  disregarded  since  they  did  not  have  any  bearing  towards 
the  success  or  failure  of  the  research.  This  subtask’s 
purpose  was  to  introduce  the  various  systems  and  verify  their 
correct  operation. 


Figure  24.  Digitized  Image  of  Target  Towards 
Edge  of  Work  Frame 


In  summary,  this  subtask  introduced  and  demonstrated  the 
various  hardware  and  software  used  throughout  the  research 
effort.  The  results  demonstrated  the  ability  to  successfully 
implement  an  open  loop,  static  look-and-move  configuration  to 
point  to  the  centroid  of  a  target  placed  anywhere  in  the  work 
frame . 

The  results  presented  next  address  the  essence  of  the 
research  effort. 

Subtask  2 :  Camera  Mounted  on  Arm ;  Tracking 

Subtask  2  initiated  the  first  step  towards  developing  an 
integrated,  closed  loop,  static  look-and-move  visual  control 
system.  This  subtask  enabled  the  VRS  to  track  any  target 
placed  anywhere  in  the  camera’s  FOV .  The  camera  was  mounted 
to  the  third  joint  of  the  PUMA  560  and  software  was  written 
to  guide  the  movement  of  the  arm  based  on  visual  feedback 
received  from  the  vision  system. 

The  vision  system  was  calibrated  (using  the  procedures 
outlined  by  the  Univision  User’s  Manual)  from  an  initial 
starting  position  based  on  robot  joint  positions, 

#start . position .  The  initial  starting  position  selected  for 
this  research,  aligned  the  camera  perpendicularly  to  the 
floor.  The  starting  position  was  stored  in  software, 
allowing  the  VRS  to  start  from  the  same  point  throughout 
testing  and  development.  A  vision-robot  frame  did  not  have 
to  be  established  as  in  Subtask  1  because  the  camera  was 
mounted  parallel  to  the  robot’s  end-effector.  This 


configuration  considered  the  robot  end-effector’s  current 
location  to  be  the  center  of  the  camera’s  FOV  translated 
along  the  camera’s  viewing  axis. 

Once  a  target  (white  ball)  was  placed  in  the  camera’s 
FOV,  the  program  track. targ  was  executed.  The  program 
instructed  the  vision  system  to  take  a  picture  and  provide 
the  two-dimensional  position  (considered  an  offset)  of  the 
target’s  centroid.  The  program  corrected  the  offset  by 
commanding  the  robot  to  move  by  an  amount  equal  to  the 
offset.  The  program  repeated  itself  until  the  user  aborted 
it,  therefore,  enabling  the  system  to  maintain  track  on  the 
target  as  the  target  moved.  It  should  be  noted  that  the  use 
of  a  high  contrast  target,  i.e.,  a  white  ball  against  a  black 
background  always  guaranteed  easy,  unambiguous  "target" 
location.  The  general  task  of  target  location  (finding  an 
arbitrary  target  in  an  arbitrary  scene)  was  not  considered 
part  of  this  research. 

Robot  motion  continued  until  the  vision  system 
determined  that  the  target  was  within  a  window  of  +  10  mm 
from  the  center  of  the  camera’s  FOV.  Ten  millimeters  was 
selected  to  reduce  the  amount  of  time  required  to  center  the 
target  in  the  camera’s  FOV.  Without  any  window,  however,  the 
VRS  oscillated  continuously  around  the  target,  primarily 
because  of  jitter  in  the  video  quantizer  and  robot 
positioning  system.  Since  the  application  for  this  thesis 
involved  the  refueling  of  an  aircraft  through  a  refueling 


50 


port  that  is  compliant  within  a  30  degree  cone  of  insertion, 
the  use  of  a  tolerance  window  should  not  be  a  problem.  It 
was  observed  that  for  this  task,  using  the  previously 
described  equipment,  the  time  for  the  system  to  center  itself 
on  a  stationary  target  (white  ball),  place  on  one  corner  of 
the  camera's  FOV,  took  approximately  2.0  seconds  ( «  0 . 7 
seconds  was  used  in  the  image  processing).  Figure  25  shows 
the  sequential  motion  of  the  digital  images  on  the  video 
monitor  as  the  VRS  visually  servos  itself  to  the  target. 


Figure  25.  Sequential  Motion  During  Visual  Servoing 

The  oscillation  mentioned  above  was  due  to  errors 
associated  with  the  calculation  of  robot  transformations  and 
with  the  image  processing  of  the  target.  The  errors  associated 
with  the  calculation  of  the  desired  final  location  in  the  robot 
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controller  occurred  "whenever  a  transformation  is  used  to 
define  the  destination  of  a  robot  motion"  (17).  The 
transformation  must  be  converted  into  individual  joint 
positions,  inevitably  inducing  (small)  errors  (17). 

Factors  that  can  affect  accurate  image  processing  and 
thus  cause  errors  in  measurements  include: 

1)  Image  contrast,  intensity,  and  threshold 

2)  Focus  and  parallax 

3)  Vision  calibration 

4)  Video  noise/spatial  quantization. 

Most  of  the  errors  introduced  by  these  factors  were  minimized 
by  using  a  white  ball  for  the  target  and  a  black  background. 
The  voltage  threshold  for  the  video  system  was  set  to  the 
middle  of  the  threshold  scale,  which  was  the  optimal  value  as 
discussed  in  the  Univision  User’s  Manual  (18).  This  provided 
the  maximum  contrast  and  intensity  during  the  research.  A 
ball  also  provided  a  circular  image  from  any  viewing  angle 
even  if  the  lens  was  not  in  perfect  focus.  Parallax  induced 
errors  were  ignored  since  the  VRS  eventually  aligned  the 
camera  above  the  center  of  the  target.  As  stated  earlier, 
the  VRS  was  calibrated  at  an  initial  start  position,  thus 
minimizing  any  calibration  errors.  The  video  noise/spatial 
quantization  errors  are  random  and  related  to  target  size  and 
the  related  image  boundary  perturbations  and  can  be  ignored 
due  to  the  tolerance  window. 

Additionally,  the  vision  system  coordinates  were  scaled 
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by  the  scale  factor  determined  during  vision  calibration. 

With  the  camera  mounted  to  the  arm,  it  was  observed  that  the 
positive  y  direction  of  the  camera’s  FOV  was  the  negative  y 
direction  of  the  robot’s  coordinate  system.  It  was  also 
observed  that  the  y  component  provided  by  the  vision  system 
had  to  be  further  scaled  by  a  factor  of  2/3’s  to  enable 
accurate  positioning  of  the  vision-robot  system.  This 
additional  scale  factor  was  necessary  to  account  for  the 
difference  in  the  x  and  y  dimensions  of  each  pixel  element, 
which  is  dependent  on  camera  type. 

Finally,  to  maintain  active  PD  control,  the  robot  motion 
instructions  for  this  task  were  broken  up  into  no  less  than 
10  continuous  "back-to-back"  move  commands  as  opposed  to  1  or 
2  move  commands.  With  less  than  10  move  commands,  it  was 
observed  that  the  VRS  would  continuously  overshoot  the 
target.  The  overshoot  problem  resulted  from  the  same  errors 
as  those  mentioned  above  for  the  oscillations  and  also  during 
the  deceleration  of  the  arm  as  it  moved  towards  its  desired 
final  location.  The  deceleration  induced  errors  are  due  to 
the  P  feedback  control  loop  taking  over  control  of  the  PUMA. 
Breaking  up  the  move  commands  allowed  smooth  robot  motion 
with  minimal  overshoot  of  the  target. 

In  summary,  the  results  of  this  subtask  demonstrated  the 
ability  of  the  VRS  to  successfully  implement  closed  loop, 
static  look-and-move  visual  servo  control  techniques  to  track 
a  white  ball  placed  anywhere  in  the  camera’s  FOV. 
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Subtask  3 :  Camera  on  Arm ;  Tracking  and  Acquiring 

Subtask  3  continued  the  work  accomplished  in  Subtask  2 
by  enabling  the  VRS  to  move  towards  its  target  as  well  as 
track  it. 

To  provide  the  VRS  with  the  knowledge  of  how  far  to  move 
towards  the  target,  the  vision  system  was  trained  to 
recognize  a  target  at  a  predetermined  distance.  To  enable 
quick  image  processing,  only  the  area  of  the  target  was  used 
for  identification  (another  reason  for  using  a  ball,  because 
its  two-dimensional  image  is  always  a  circle).  From  the 
starting  position  defined  in  Subtask  2,  it  was  determined 
that  joint  6  of  the  PUMA  560  was  approximately  350  mm  from 
the  floor.  From  this  initial  position,  the  VRS  was  trained 
to  the  white  ball  target.  This  initial  target,  referred  to 
as  Target  1.  was  assigned  to  the  first  of  nine  locations  in 
the  vision  system’s  memory.  Each  subsequent  location  in 
memory  was  trained  with  the  target  25  mm  closer  than  the 
previous  target.  Therefore,  if  Target  1  was  identified  by 
the  vision  system,  joint  6  would  be  350  mm  from  the  target 
and  150  mm  from  the  target  if  Target  9  was  identified. 

Once  the  target  was  placed  in  the  camera’s  FOV ,  the 
program  track . bal 1 . scat  was  executed.  The  program 
accomplished  the  same  tasks  as  track. targ  with  the  addition 
of  having  the  vision  system  identify  the  target  (based  on  the 
target’s  area).  The  program  used  this  additional  information 
to  instruct  the  VRS  to  move  towards  the  target.  Robot  motion 
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continued  until  the  target  was  at  the  desired  distance  away 
from  the  VRS  and  within  the  10  mm  window  discussed  in  Subtask  2. 

As  stated  previously,  the  camera  was  mounted  on  the 
third  joint  of  the  PUMA  560.  The  PUMA’s  controller  defined 
the  robot’s  end  position  at  its  sixth  joint  with  respect  to 
the  robot  base.  It  also  doesn’t  use  just  the  first  three 
joints  to  specify  position  when  solving  the  inverse  kinematic 
solution  for  the  final  joint  locations.  This  was  not  a 
problem  in  Subtask  2,  where  only  tracking  of  a  target  was 
required.  However,  in  the  subtask  described  here,  it  was 
found  that  the  VRS  lost  camera  sight  (two-dimensional)  of  the 
target  as  the  VRS  approached  the  target.  Therefore,  an 
offset  had  to  be  programmed  to  account  for  the  position  of 
the  camera  as  the  camera  approached  the  target.  This  offset 
correction  enabled  the  VRS  to  successfully  track  and  approach 
the  target. 

To  further  minimize  the  positional  errors 
encountered  during  the  VRS ’ s  approach  to  the  target,  the 
program  instructed  the  VRS  to  converge  towards  the  target  in 
increments.  This  distance  convergence  approach  proved 
successful  and  it  was  observed  that  the  time  for  the  VRS  to 
approach  and  center  itself  above  a  stationary  target,  placed 
at  one  corner  of  the  camera's  FOV ,  took  approximately  5.6 
seconds.  Figures  26,  27,  and  28  show  the  sequential  motion 
of  the  VRS  and  of  the  digital  images  on  the  video  monitor  as 
the  VRS  visually  servos  itself  to  the  target. 
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Figure  27.  Intermediate  Position  of  VRS 
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Figure  28.  Final  Position  of  VRS 


Originally,  the  VRS  was  to  move  within  50  mm  of  the 
target.  However,  oscillations  occurred  whenever  the  target’s 
image  in  the  camera’s  FOV  encompassed  an  area  greater  than  or 
equal  to  half  the  viewing  area  of  the  camera.  The 
rise  i  ]  1  a  t  i  ons  were  due  to  the  video  noise/spatial  quantization 
and  the  transformation  conversion  errors  discussed  in  Sub¬ 
task  2.  The  situation  was  resolved  by  allowing  the  VRS  to 
approach  the  target  no  closer  than  200  mm.  Since  the 
application  of  this  thesis  involves  the  refueling  of  an 
aircraft,  the  limited  approach  does  not  pose  a  problem.  As 
shown  in  the  artist’s  conception  of  the  robotic  refueler  in 


Figure  29,  the  camera  was  mounted  to  the  refueler  s  third 
joint,  with  the  final  location  of  the  camera  positioned  above 
and  away  from  the  refueling  port,  during  the  refueling 
operation . 


Additional  work  accomplished  in  Subtask  3  enabled  the 
VRS  to  become  a  tracker  if  the  target  started  to  move  around 
in  the  camera’s  FOV .  This  was  an  unlikely  event  for  an 
application  involved  with  refueling  a  stationary  aircraft, 
however,  it  could  be  useful  for  other  applications  where  the 
target  is  allowed  to  move. 

In  summary,  the  results  of  this  subtask  demonstrated  the 
ability  of  the  VRS  to  successfully  implement  closed  loop, 
static  look-and-move  visual  servo  control  techniques  to  track 
and  approach  a  white  ball  placed  anywhere  in  the  camera’s 
FOV  . 

Subtask  4 :  Camera  on  Arm ;  Search 

Subtask  4  completed  the  overall  task  of  developing  an 
integrated  VRS  that  searched  for  and  acquired  the  desired 
target  . 

Once  the  target  was  placed  anywhere  in  a  predetermined 
"search  area",  the  program  search  was  executed.  The  program 
instructed  the  VRS  to  search  for  the  target.  Once  the  target 
was  found,  Subtask  3  took  over  and  visually  servoed  the  VRS 
to  the  target . 

The  search  pattern  for  this  subtask  was  divided  into  16 
increments  in  the  robot’s  WCS  x-y  plane.  The  increments  were 
determined  by  measuring  the  window  (perimeter)  visible  in  the 
camera’s  FOV.  For  the  specific  CRT  display  used  in  this 
project,  the  size  of  the  window  was  approximately  165  mm  in 
the  x  direction  by  135  mm  in  the  y  direction.  An  overlap 


59 


between  the  increments  in  the  search  pattern  was  added  to  the 
program,  search,  to  ensure  the  entire  search  area  was 
scanned.  Increments  of  80  mm  were  used  when  the  VRS  moved  in 
the  positive  or  negative  x  direction  and  in  increments  of  65 
mm  in  the  positive  or  negative  y  direction.  The  entire 
search  covered  an  area  of  approxi ma tely  400  mm  by  380  mm , 
this  was  limited  by  the  work  envelope  of  the  PUMA.  The  time 
required  to  find  the  target  depended  on  where  the  target  was 
placed  in  the  search  area.  The  search  time  ranged  from  0.2 
seconds  directly  below  the  VRS  to  22.0  seconds  when  the 
target  was  placed  in  the  center  of  the  search  area.  Figures 
30,  31,  and  32  show  only  three  of  the  sequential  motions  of 
the  VRS  as  it  searched  for  the  target  and  then  servoed 
towards  it. 


Figure  30.  Starting  Position  of  VRS 
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Figure  31. 


Intermediate  Position  of  VRS 


Figure  32.  Final  Position  of  VRS 


During  testing  of  this  subtask,  it  was  discovered  if  the 
target's  initial  position  in  the  search  area  was  at  a 
distance  greater  than  that  for  which  the  vision  system  was 
trained,  the  target  could  not  be  identified.  This  did  not 
prevent  the  vision  system  from  acquiring  the  target  since 
acquisition  was  effected  by  previously  existing  detection  and 
centroid  computation  inherent  to  the  vision  system.  When  the 
search  mode  was  initiated,  the  vision  system  decided  that  a 
target  was  in  the  camera’s  FOV  when  its  area  (in  pixels)  was 
greater  than  the  smallest  acceptable  blob  area  set  by  the 
user.  For  this  research,  the  minimum  area  was  arbitrarily 
set  to  10  pixels.  To  enable  target  verification  following 
acquisition,  the  VRS  approached  the  target  in  25  mm 
increments  until  the  vision  system  could  identify  the  target. 
Once  the  target  was  successfully  identified,  the  VRS  visually 
servoed  itself  to  the  target  as  in  Subtask  3. 

Additional  work  accomplished  in  Subtask  4  enabled  the 
VRS  to  automatically  restart  the  search  if  the  target  was 
ever  removed  from  the  camera’s  FOV.  This  was  an  unlikely 
event  for  an  application  involved  with  refueling  a  stationary 
aircraft,  however,  it  could  be  useful  for  other  applications 
where  the  target  is  allowed  to  move. 

In  summary,  the  results  of  this  subtask  demonstrated  the 
ability  of  the  VRS  to  search  for  a  target  and  successfully 
implement  closed  loop,  static  look-and-move  visual  servo 
control  techniques  to  track  and  approach  the  target. 
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The  combined  results  of  the  four  subtasks  described 


above  achieved  the  desired  goal  of  this  research.  The  next 
section  discusses  the  results  from  additional  research  which 
implemented  a  closed  loop,  dynamic  look-and-move  visual 
control  scheme  in  a  tightly  controlled  environment. 

Dynamic  Look-And-Move  System 

The  task  of  developing  a  closed  loop,  dynamic  look-and- 
move  visual  servo  control  system  was  an  extension  of  the  work 
accomplished  in  Subtask  3  of  the  Static  Look-and-Move 
Task.  However,  dynamic  visual  servo  control  was  implemented 
when  the  VRS  moved  towards  the  target. 

Once  the  target  was  placed  near  the  center  of  the 
camera’s  FOV,  the  program  track . targ . dcat  was  executed.  From 
all  indications,  the  VRS  performed  in  the  same  manner  as  in 
Subtask  3.  However,  a  difference  existed  in  the  movement  of 
the  VRS  towards  the  target.  In  this  task,  the  robot  motion 
instructions  were  broken  up  into  only  2  continuous  "back-to- 
back"  move  commands  as  opposed  to  the  10  move  commands  in 
Subtask  3.  Also  robot  motions  were  instructed  to  occur  at 
half  the  speed  of  those  in  Subtask  3.  Under  the  concept  of 
procedural  motion,  a  picture  was  taken  during  the  second  move 
command  in  the  program.  The  data  from  the  image  provided  an 
in-course  correction  to  the  VRS,  thus  allowing  a  smoother 
approach  to  the  target. 

A  limitation  existed  in  this  task  because  of  the 
oscillation  problem  discussed  in  Subtask  2.  Therefore,  the 
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target  had  to  be  placed  near  the  center  of  the  camera’s  FOV 
for  the  VRS  to  work  correctly.  It  was  observed  for  this 
task,  that  the  time  for  the  VRS  to  approach  and  center  itsel 
above  a  stationary  target  took  approximately  6.1  seconds. 
This  is  a  slower  time  than  Subtask’s  3,  however,  as  stated 
above,  robot  motions  were  at  half  the  speed.  When  Subtask  3 
was  executed  at  the  slower  speed,  it  took  approximately  7.8 
seconds  for  the  VRS  to  approach  and  center  itself  above  the 
stationary  target.  Figures  33,  34,  and  35  show  the 
sequential  motions  of  the  VRS  as  it  dynamically  servoed 


Figure  33.  Starting  Position  of  VRS 


This  completes  the  presentation  of  the  results  of  the 


research  effort,  the  conclusions  and  recommendations  are  now 
presented . 
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V. 


Conclusions  and  Recommendations 


In  this  research,  visual  servo  control  schemes  were 
designed  and  integrated  for  a  PUMA  560  robot  arm  which 
derived  its  visual  information  from  a  MIC  vision  system. 

The  vision  system’s  camera,  with  fixed  focused  lens,  was 
rigidly  mounted  to  the  third  joint  of  the  PUMA.  The  two 
control  schemes  researched  involved  static  and  dynamic,  look- 
and-move  visual  control  techniques.  These  techniques 
successfully  implemented  and  demonstrated  the  concept  of 
using  a  robot,  equipped  with  a  vision  sensor  system,  to 
ground  refuel  an  aircraft. 

Conclusions 

Two  conclusions  were  drawn  from  the  success  of  this 
research  effort.  First,  the  results  presented  in  Chapter  IV 
proved  that  the  closed  loop,  static  1 ook -and-move  visual 
servo  control  vision-robot  system  ( VRS )  could  successfully 
find  and  move  towards  a  well  defined  target,  a  white  ball, 
placed  in  a  designated  search  area.  The  vision  system  was 
used  to  identify  the  target  and  determine  its  two-dimensional 
centroid  position  with  respect  to  the  robot’s  end-effector. 
Because  one  camera  with  a  fixed  focus  lens  was  used,  a  crude 
method  was  developed  to  determine  the  distance  from  the 
target  to  the  camera.  As  the  camera  approached  the  target, 
the  target’s  image  grew  larger  with  respect  to  the  camera’s 


67 


field  of  view  ( FOV ) .  Therefore,  by  training  the  target  to 
the  vision  system  at  different  distances,  the  vision  system 
could  determine,  by  which  target  was  identified,  the  distance 
from  the  target  to  the  VRS .  This  distance  measurement  along 
with  the  centroid  position  measurement  provided  three- 
dimensional  information  about  the  target.  This  method  worked 
adequately  for  a  well  defined  target  like  a  ball,  however,  an 
improved  method  would  be  required  for  a  more  complex  target. 

Finally,  the  results  from  Chapter  IV  also  proved  closed 
loop,  dynamic  1 ook-and-move  visual  servo  control  techniques 
could  be  successfully  implemented.  However,  due  to  slow 
computer  processing  and  the  inability  to  implement  parallel 
processing,  limitations  existed  with  this  control  technique. 
These  limitations  were  overcome  by  using  procedural  motion. 
During  movements  of  the  VRS  over  large  distances,  dynamic 
visual  control  could  only  be  successfully  implemented.  This 
situation  only  occurs  when  the  VRS  moves  towards  the  target. 
Thus,  although  limitations  existed  in  the  closed  loop, 
dynamic  1 ook-and-move  visual  control  technique,  excellent 
results  were  achieved  when  the  target  was  initially  placed 
near  the  center  of  the  camera's  FOV. 
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Recommendations 

The  work  accomplished  in  this  research  effort  provides  a 
solid  foundation  on  which  to  continue  work  in  areas  of 
robotics  and  vision.  The  following  are  recommendations  for 
continued  research  in  areas  related  to  this  thesis: 

-  Implement  the  visual  servo  control  techni ques 
presented  in  this  thesis  on  the  Robotics  and  Automation 
Laboratory  ( RAL )  Hierarchical  Control  System  (RHCS)  (9).  The 
RHCS  will  be  the  primary  system  used  for  research  with  the 
PUMA  560  at  the  Air  Force  Institute  of  Technology  (AFIT). 

The  RHCS  also  allows  the  user  to  specify  which  joints  are 
controlled.  This  alleviates  the  problem  associated  with 
solving  for  position  using  inverse  kinematics. 

-  Using  the  RHCS,  develop  a  parallel  processing 
capability,  to  enable  real-time,  closed  loop,  dynamic  look- 
and-move  visual  servo  control  for  all  robot  motions. 

-  Decrease  the  computer  processing  time  involved  during 
inter-computer  communication  by  developing  a  quicker,  more 
efficient  method  for  transferring  messages  between  the  two 
computers.  Almost  twice  as  much  time  was  spent  on 
communication  than  image  processing  (about  0.5  sec.  for 
communication  and  0.3  sec.  for  image  processing  of  the  ball 
target).  A  total  processing  time  of  less  than  .1  seconds  is 
required  for  real-time  visual  servo  control  (14:214). 
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-  Implement  various  image  processing  techniques 
currently  in  existence  at  AFIT  to  enable  the  acquisition  of 
more  visually  complex  targets  against  an  arbitrary 
background . 

-  Implement  a  more  efficient  method  to  determine  the 
distance  to  a  target.  Various  techniques  include  using  the 
following:  auto  focus  camera;  stereo  vision;  or  sonar. 

-  Remount  the  camera  on  joint  three  at  an  angle,  such 
that  the  center  of  the  camera’s  FOV  will  be  the  at  the  center 
of  the  robot’s  sixth  joint,  only  translated  along  the  z  axis. 
This  will  alleviate  the  requirement  for  programming  an  offset 
for  the  camera  location. 

-  Obtain  a  camera  which  weighs  less  then  5  kgs  and  mount 
it  to  the  sixth  joint.  This  would  be  valuable  to 
applications  where  the  target’s  orientation  is  important. 
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Appendix  A 


Equipment  and  Interface  Descriptions 

This  Appendix  contains  brief  descriptions  of  the 
hardware  and  software  used  in  this  research  effort.  Part  1 
contains  sections  of  Chapter  1  from  (16)  which  is  an 
introduction  and  description  of  the  PUMA  Mark  II,  500  Series, 
Robot  System  (in  particular  the  PUMA  560).  Part  2  is  a  copy 
of  the  prefatory  information  from  (18)  which  provides  an 
introduction  and  description  of  the  Machine  Intelligence 
Corporation’s  (MIC)  Vision  System.  Finally,  Part  3  is  a  copy 
of  (19)  which  contains  descriptions  of  the  interface  software 
between  the  MIC  Vision  System  and  Unimations’  VAL  II 
programming  language. 

A  miniature  Table  of  Contents  follows: 

Table  of  Contents 


Part  Page 

1  .  Description  of  PUMA  560  .  72 

2.  Description  of  MIC  Vision  System  .  92 

3.  Interface  Software  Descriptions  .  106 
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Part  1 


Description  of  PUMA  560  (16) 
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CHAPTER  1 


INTRODUCTION  AND  DESCRIPTION 


1-1.  INTRODUCTION 

The  Unimate  PUMA  Mark  II,  500  Series  Robot  (Figure  1-1),  is  an 
advanced  computer  controlled  robot  arm  system,  manufactured  by 
Unimation  Incorporated,  A  Wes t i nqhouse  Company,  of  Danbury, 

Connect  icut . 

This  equipment  manual  is  to  be  used  with  VAL  II  and  VAL  PLUS 
operating  systems.  It  contains  the  introduction  and  description, 
installation,  operation,  maintenance,  troubleshooting,  spare 
narts  list,  appendixes,  and  a  glossary. 

Instructions  on  how  to  write  and  execute  proqrams  for  the  PUMA 
Mark  II  system  is  given  in  the  "User's  Guide  To  VAL  II,"  Part  No. 
398T1,  or  "User's  Guide  to  VAL  PLUS,"  Part  No.  398AC1. 

Note 

Appropriate  proqramminq  manuals  will  be 
sent  to  users  of  VAL  II  or  VAL  PLUS  opera¬ 
ting  systems. 

The  mechanical  and  electrical  drawings  are  contained  in  two  separ¬ 
ate  manuals  as  follows: 

PUMA  Mark  II,  500  Series,  Mechanical  Drawing  Set,  394AB1 
(for  VAL  II  or  VAL  PLUS  operating  systems). 

PUMA  Mark  II,  500  Series,  Electrical  Drawing  Set,  394AC1 
(for  VAL  II  or  VAL  PLUS  operating  systems). 

Please  note  that  personnel  responsible  for  programming  and  opera¬ 
ting  the  PUMA  are  expected  to  attend  the  training  course  given  by 
the  Unimation  Technical  Training  Department.  They  must  also  have 
complete  understanding  of  all  information  given  in  this  manual 
and  the  User's  Guide.  The  combined  information  from  the  training 
program  and  manuals  will  ensure  safe  and  efficient  operation  of 
the  PUMA  robot. 

1-2.  DESCRIPTION  -  PUMA  SYSTEM  AND  SOFTWARE 

1-2-1.  PUMA  SYSTEM.  The  PUMA  robot  system  is  designed  to  adapt 
to  a  wide  ranqe  of  applications.  The  basic  units  are  the  teach  pend 
ant,  software,  controller,  peripherals,  and  robot  arm. 
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The  system  software  that  controls  the  robot  arm  is  stored  in  the 
computer  memory  located  in  the  controller,  which  also  houses  the 
operating  controls  for  the  system. 

To  teach  the  robot  arm,  either  of  two  procedures  can  be  used. 

The  teach  pendant  may  be  used  to  manually  direct  the  movements  of 
the  robot  arm  through  each  step  of  the  task.  These  steps  are  re¬ 
corded  and  then  stored  in  the  computer  memory.  The  second  method 
is  to  write  a  program  using  software  instructions.  Position  data 
and  software  programs  are  entered  into  the  computer  memory 
through  the  peripheral  terminal  keyboard  or  throuqh  the  teach 
pendant . 

In  either  case,  the  controller  transmits  the  instructions  from 
the  computer  memory  to  the  arm.  Position  data  obtained  from 
incremental  encoders  and  potentiometers  in  the  robot  arm  are 
transmitted  back  to  the  controller/computer  to  provide  closed- 
loop  control  of  the  arm  motions. 

The  programs  may  also  be  stored  external  to  the  controller  on  a 
floppy  disk,  a  small  flexible  disk  coated  with  a  maqnetic  medium 
that  provides  a  permanent  record  of  the  program.  The  floppy  disk 
unit  is  an  optional  peripheral. 

Additionally,  the  PUMA  can  be  programmed  to  interact  with  its 
environment  by  using  external  input  and  output  signals.  External 
input  (WX)  signals  can  be  used  to  halt  a  program,  branch  to  another 
program  step,  or  branch  to  another  subroutine.  For  example,  an  ex¬ 
ternal  input  signal  can  stop  the  program  when  safe  operation  is 
impaired.  External  output  (OX)  signals  allow  the  PUMA  system  to 
control  other  equipment  related  to  its  work  environment.  If  the 
WX/OX  features  are  used,  at  least  one  optional  I/O  module  is  re- 
qu  i  red . 

The  robot  arm  executes  the  instructions  transmitted  to  it  by  the 
controller.  The  arm  assembly  is  capable  of  rotational  movement 
around  six  axes  (five  axes  for  the  PUMA  550).  The  axes  of  rotation 
are  shown  in  Figure  1-2. 

1-2-2.  SOFTWARE.  The  PUMA  system  operates  on  a  high  level  lan- 
quaqe  called  VAL  II  or  VAL  PLUS.  In  addition  to  being  a  sophis¬ 
ticated  programming  language  developed  for  assembly,  it  is  a  com¬ 
plete  robot  control  system. 


1-2 
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WAIST  (JOINT  1) 


WAIST  (JOINT  1) 


Figure  1-2.  Robot  Arm:  Joint  Identification 
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The  VAL  II  or  VAL  PLUS  proqramminq  language  consists  of  a  full 
set  of  English  language  instructions  for  teaching  and  editing. 
However,  the  VAL  II  control  system  has  additional  capabilities 
that  readily  communicate  with  other  computer  based  systems  such 
as  vision  and  force  sensors,  as  well  as  with  supervisory  computer 
systems . 

Robot  proqramminq  can  be  accomplished  by  the  teach-by-show i nq 
method  using  the  teach  pendant  or  by  keyboard  entry.  The  full 
proqramminq  versatility  can  be  realized  only  throuqh  the  key¬ 
board  . 

In  either  proqramminq  method,  all  tauqht  points  are  stored  as 
transformations  (referenced  to  a  coordinate  system  fixed  relative 
to  the  stationary  robot  base),  as  precision  points  (position 
information  stored  in  the  form  of  ioint  anqles),  or  as  compound 
transformations  (point  locations  referenced  to  previous  locations 
as  a  measurement  from  a  Cartesian  coordinate  system  fixed 
relative  to  the  tool  mounting  surface). 


1-3.  DESCRIPTION  -  CONTROLLER 

The  controller  is  the  master  component 
All  signals  to  and  from  the  robot  pass 
are  used  by  it  to  perform  real-time  ca 
movement  and  position  (Figure  1-3).  ( 
discussed  in  paragraph  1-4.) 


of  the  electrical  system, 
throuqh  the  controller  and 
lculations  to  control  arm 
Peripheral  components  are 


Operatinq  controls  and  indicators  are  located  on  the  front  and 
top  panels  of  the  controller.  Connections  for  the  robot  arm, 
terminal,  floppy  disk  drive,  I/O  modules,  and  accessory  are 
located  on  the  controller  rear  panel.  Software  is  stored  in  the 
computer  memory,  located  in  the  controller.  The  software 
interprets  the  operating  instructions  for  the  robot-  arm,  and  the 
controller  transmits  these  instructions  from  the  computer  memory 
to  the  arm.  From  incremental  encoders  and  potentiometers  in  the 
robot  arm,  the  controller/computer  receives  data  about  arm 
position.  This  provides  a  closed  loop  control  of  arm  motions. 


1-4 
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The  main  internal  components  of  the  controller  are  listed  below. 
Their  locations  are  shown  in  Fiqures  1-4  and  1-4A,  and  described 
in  paragraphs  1-3-1  throuqh  1-3-9. 

a.  DEC  LSI-11  computer  (DEC  and  LSI-11  are  trademards  of 

Digital  Equipment  Corporation).  (See  Note.) 

b.  DLV11-J  quad  serial  interface  boards  (two).  (See  Note.) 

c.  CMOS  board.  (See  Note.) 

d.  "A"  interface  board  (with  boot  chips) 

e.  "B"  interface  board  (with  clock) 

f.  Digital  servo  boards  (six) 

g.  Power  amplifier  assemblies  (one  major  and  one  minor) 

h.  Power  amplifier  control  board 

i.  Input/output  interface  board  (not  shown) 

j.  Power  supplies  (two) 

k.  Hiqh  power  function  board 

l .  Arm  cable  board 

Note 

1.  The  VAL  PLUS  operating  system  contains  an 
LSI-11/23  processor,  48K  CMOS,  (additional 
memory  optional),  and  one  quad  serial  board. 

2.  The  VAL  II  operating  system  contains  an 
LSI-11/73  processor,  64K  CMOS  (additional 
memory  optional),  and  two  quad  serial  boards. 

1-3-1.  LSI-11  COMPUTER  SYSTEM.  The  LSI-11  system  is  a  stand¬ 
ard  DEC  unit  containing  a  processor,  memory  and  communication 
boards.  System  software  and  user  programs  are  stored  in  a 
Complimentary  Metal  Oxide  Semiconductor  (CMOS)  nonvolatile  memory. 
Communication  between  processor  and  other  components  is  accom¬ 
plished  as  follows: 

a.  A  four-oort  asynchronous  serial  interface  board 
(DLV11-J)  links  the  processor,  terminal,  teach  pendant,  and  high- 
density  floppy  disk.  A  second  DLVII-J  board  is  available  to  serve 
as  the  link  between  the  system  processor  and  a  Supervisory  pro¬ 
cessor  system.  This  same  board  provides  communications  along 
the  real-time  trajectory  modifications  link  known  as  ALTER. 
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Figure  1-4.  Controller:  Board  Location  (Top) 
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Fiqure  1-4A. 


Controller:  Board  Location  (Bottom) 


1).  The  "A"  interface  board  provides  a  communications  link 
to  the  “B"  interface  board  (paraqraph  1-3-2).  The  analog-to- 
diqital  converter  that  reads  the  potentiometers  is  located  on 
the  "A"  interface  board. 

1-3-2.  "B"  INTERFACE  BOARD.  The  "B"  interface  board  links  the 
LSI-11  system  throuqh  the  "A"  interface  board,  to  the  servo  drive 
side  of  the  control  system.  Command  signals  sent  by  the  LSI/11 
are  interfaced  to  a  servo  bus  by  this  board.  Once  the  commands 
or  data  have  been  interpreted  and  acted  upon,  return  signals  are 
sent  to  the  LSI-11  throuqh  the  "A"  interface  board.  Multiplexer 
circuitry  on  this  board  provides  analoq-to-d iq i ta 1  conversion  of 
motor  Dotent iometer  feedback  and  clock/terminator  functions. 

1-3-3.  DIGITAL  SERVO  BOARDS.  There  are  six  diqital  servo 
boards,  one  for  each  joint.  Each  joint  is  controlled  by  a 
separate  microprocessor.  Position  inputs  from  the  computer,  as 
dictated  by  the  LSI-11  calculations,  are  fed  into  the  diqital 
servo  board  every  28  milliseconds.  Diqital  information  is  fed  to 
a  diq i ta 1-to-analoq  convertor  to  qenerate  the  analoq  signal 
required  to  drive  the  DC  servomotors.  Communications  from  each 
diqital  servo  board  through  the  "A"  and  "B"  interface  boards,  to 
the  LSI-11,  are  controlled  by  a  protocol  using  interrupt  servic¬ 
ing  routines. 

1-3-4.  POWER  AMPLIFIER  ASSEMBLIES.  For  each  joint,  the  output 
from  the  diqital  servo  board  is  fed  to  a  power  amplifier  board, 
where  it  is  amplified  to  voltage  and  current  levels  high  enough 
to  drive  the  servomotors.  Outputs  from  the  power  amplifier 
boards  are  fed  to  the  arm  through  the  connecting  cable  via  the 
arm  cable  board  assembly  (paragraph  1-3-10). 

1-3-5.  POWER  AMPLIFIER  CONTROL  BOARD.  The  power  amplifier 
control  board  contains  the  following: 

a.  Six  joint  error  indicator  lights.  When  lit,  these  lights 
indicate  an  overcurrent  hardware  failure. 

b.  A  joint  reset  pushbutton.  Pressing  pushbutton  resets  joint 
after  error  has  been  corrected. 

c.  Two  hiqh  temperature  indicator  liqhts,  Thl  and  Th2.  When 
lit,  these  lights  indicate  the  oresence  of  high  temperature  at  the 
power  amplifier  assemblies. 
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1-5.  DESCRIPTION  -  ROBOT  ARM 

The  robot  arm  is  the  mechanical  component  of  the  system  incorpo¬ 
rating  6  deqrees  of  freedom  (5  degrees  on  PUMA  550),  each  con¬ 
trolled  by  a  DC  servomotor.  It  is  sufficiently  flexible  to  be 
taught  a  wide  variety  of  tasks.  Each  member  of  the  robot  arm  is 
connected  to  another  member  at  a  ioint,  much  like  a  human  arm  and 
torso.  Throuqh  each  joint  passes  one  or  more  axes  around  which 
the  members  of  the  arm  rotate. 

The  members  of  the  robot  arm  are  shown  in  Fiqure  1-15;  they  are 
the  trunk,  shoulder,  upper  arm,  forearm,  wrist,  and  qripper. 

The  robot  arm  members  contain  the  various  servomotors  and  qear 
trains. 

To  achieve  maximum  strength  with  minimum  weight,  the  upper  arm 
and  forearm  are  of  monocoque  contruction.  Monocoque  is  a  method 
of  construction  that  uses  the  covering  plates  or  "skin"  of  an 
assembly  to  carry  all  or  part  of  the  stresses. 

The  axes  of  rotation  and  the  ranges  of  rotation  are  shown  in 
Figure  1-16  and  described  in  Table  1-3. 
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WAIST  320° 


WAIST  320° 


Figure  1-16.  Robot  Arm:  Degrees  of  Joint  Rotation 
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Table  !-3.  Robot  Arm  Axes 


JOINT 

DESCRIPTION 

Waist  -  Joint  1 

Joint  1  axis  is  perpendicular  to  the 
mounting  plane  of  the  PUMA  and  coin¬ 
cident  with  the  centerline  of  the 
trunk . 

Shoulder  -  Joint  2 

Joint  2  axis  is  perpendicular  to  and 
intersects  Joint  1  axis;  it  is  coin¬ 
cident  with  the  centerline  of  the 
shoulder . 

Elbow  -  Joint  3 

Joint  3  axis  is  parallel  to  the 

Joint  2  axis. 

Wrist  -  Joint  4 
(PUMA  560  only) 

Joint  4  axis  is  perpendicular  to  and 
intersects  Joint  5  axis. 

Wrist  -  Joint  5 
(Joint  4  on 

PUMA  550) 

Joint  5  axis  (Joint  4  on  PUMA  550) 
is  parallel  to  the  Joint  2  and  3 
axes 

Wrist  -  Joint  6 
(Joint  5  on 

PUMA  550) 

Joint  6  axis  (Joint  5  on  PUMA  550) 
is  perpendicular  to  and  intersects 
Joint  5  axis  (Joint  4  on  PUMA  550); 
it  is  coincident  with  the  centerline 
of  the  gripper  mounting  flange. 

Each  member  of  the  arm  assembly  is  driven  by  a  permanent-magnet  DC 
servomotor  driving  through  its  associated  gear  train.  Each  motor 
in  the  PUMA  robot  arm  contains  an  incremental  encoder  and  a  poten¬ 
tiometer  driven  through  a  116  to  1  gear  reduction.  The  proper 
functioning  of  the  PUMA  requires  control  of  the  position  and  the 
velocity  of  each  ioint  of  the  robot  arm. 

For  a  servo-controlled  robot  system,  position  must  be  measured 
relative  to  a  known  initial,  absolute  position.  The  potentiometers, 
irtcorporated  in  the  motor,  are  used  to  initialize  the  PUMA;  that 
is,  to  establish  its  absolute  position.  The  initializing  proce¬ 
dure  must  be  done  each  time  the  PUMA  system  is  powered  up.  (Refer 
to  paragraph  3-4.) 

The  incremental  encoders  are  mounted  on  the  shaft  of  each  motor 
and  provide  position  change  and  velocity  signals  for  the  servo 
system.  Position  change  signals  are  read  from  the  encoders,  and 
velocity  signals  are  calculated.  Approximately  32  times  during 
each  28-millisecond  window  of  the  digital  servo  system,  the 
signals  from  the  encoders  are  compared  to  the  calculated  position 
and  any  necessary  correction  signals  are  generated. 
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The  servomotors  for  the  major  axes  {Joints  1,  2,  and  3)  are 
equipped  with  electroraaqnet ic  brakes.  These  brakes  are  acti¬ 
vated  when  cower  is  removed  from  the  motors,  thereby  locking 
the  robot  am  in  a  fixed  position.  This  safety  feature  removes 
the  risk  of  injury  or  damaqe  that  could  result  from  the  arm 
collapsing  it  power  is  accidentally  removed. 

Power  for  the  motors  is  supplied  throuqh  the  cable  connecting  the 
robot  arm  and  the  controller.  Feedback  siqnals  from  the  incremen¬ 
tal  encoders  and  potentiometers  are  also  carried  by  this  cable. 

1-5-1.  VJAIST  -  JOINT  1.  The  motor  for  Joint  1  is  located  out¬ 
side  the  trunk,  on  the  base  casting.  The  qear  train  is  shown  in 
Figure  1-17.  For  clarity  of  description,  the  qear  train  is  divided 
into  three  sections  (Table  1-4). 

Table  1-4  Joint  1  -  Gear  Train 


COMPONENT 

DESCRIPTION 

Motor  shaft  spur 
pinion  and  two  qears 

A  spur  pinion  cut  into  the  end  of 
the  motor  shaft  drives  two  spur 
qears  simultaneously. 

Note 

Pinion:  Of  the  two  qears  that 

run  together,  the  pinion  has 
the  smaller  number  of  teeth. 

Gear;  Of  two  gears  that 
run  together,  the  gear  has 
the  greater  number  of  teeth. 

Idler  shafts 

The  two  spur  qears  transmit  power 
throuqh  idler  shafts  that  have 
different  torsional  rigidities. 

In  one,  torsional  rigidity  is  high. 
This  prevents  any  appreciable 
twisting  of  the  shaft  around  its 
lonq  axis  (called  windup).  The 
other  shaft  has  a  lower  torsional 
rigidity  designed  for  a  predeter¬ 
mined  amount  of  windup  to  preload 
the  entire  qear  train  eliminating 
backlash . 

Two  pinions  and  the 
bull  gear 

Two  more  pinions,  at  the  end  of  the 
idler  shafts,  drive  the  bull  gear. 
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SPECIFICATIONS 


Table  1-5  contains  the  specifications  for  the  PUMA  system. 

Table  1-5.  Specifications 


ITEM 

SPECIFICATIONS 

Robot  Arm 

Axes 

Six  revolute  axes  (5  axes  for  PUMA 

550) 

Clearance  Required 

Spherical  volume  with  shoulder  at 
center:  0.92  m  (36.2  in.)  radius 

Mounting  Orientation 

Must  be  mounted  vertically.  Base 
must  be  level  within  1  degree. 

Vertical  can  mean  right  side  up  or 
upside  down  -  NOT  horizontal. 

Weight 

534. N  (120  lb) 

Drive 

Electric  DC  servomotor 

Mounting  Surface 
for  Gripper 

Four  10-24  holes  on  a  0.041  m 
(1.625  in.)  diameter  bolt  circle 

Maximum  Inertia  Load 
(Including  Standard 
Gripper:  Toggle 

Type  -  P/N  510-9013; 
Parallel  Type  -  P/N 
510-0100 ) 

Wrist  Rotation  - 
Joint  4  (PUMA  560) 

NOT  to  exceed  5.7  in.-oz-sec2  (e.g., 
5.5  lb  steel,  4  in.  dia.  disk  mounted 

5  in.  from  wrist  rotation  axis). 

Wrist  Bend  - 
Joint  4  (PUMA  550) 
Joint  5  (PUMA  560) 

NOT  to  exceed  5.7  in.-oz-sec2  (e.q., 
5.5  lb  steel,  4  in.  dia.  disk  mounted 

5  in.  from  wrist  bend  axis). 

Flange  Rotation 

Joint  5  (PUMA  550) 
Joint  6  (PUMA  560) 

NOT  to  exceed  0.5  in.-oz-sec2  (e.q., 
5.5  lb  steel,  4  in.  dia.  disk  cen¬ 
tered  on  axis  of  rotation). 

Static  Force  at  Tool 

58  N  (13.0  lb)  maximum 
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Table  1-5-  Specifications  (Cont) 


ITEM 

SPECIFICATIONS 

Position  Repeatability 

+0.1  mm  (+0.004  inches)  within  pri¬ 
mary  work  envelope  (as  measured  to 
center  of  the  tool  mounting  flange). 

Gripper  Control 

Computer  controlled,  pneumatic 

0.003  m3/s  at  710  KPa  (6  ft3/min 
at  100  psi) 

Tool  Acceleration 

1  g  maximum  (with  maximum  load) 

Tool  Velocity 

1.0  m/s  (3.3  fps)  maximum  (with  max¬ 
imum  load  within  primary  work  enve¬ 
lope  ) 

Software  Movement  Limits 

Waist  -  Joint  1 

5.59  r  (320  deg) 

Shoulder  -  Joint  2 

4.36  r  (250  deg) 

Elbow  -  Joint  3 

4.72  r  (270  deg) 

Wrist  -  Joint  4 
(PUMA  560) 

5.24  r  (300  deg) 

Wrist  -  Joint  5 
(PUMA  560) 

Wrist  -  Joint  4 
(PUMA  550) 

3.49  r  (200  deg) 

Wrist  -  Joint  6 
(PUMA  560) 

Wrist  -  Joint  5 
(PUMA  550) 

9.29  r  (532  deg) 

Joint  Anqular  Resolution 

PUMA  550/560 

Waist  -  Joint  1 
Shoulder  -  Joint  2 
Elbow  -  Joint  3 

5.7557  x  10"3  deg/bit 

4.17408  x  10-3  deg/bit 

6.7098  x  10"3  deg/bit 

PUMA  550 

Wrist  -  Joint  4 

Wrist  -  Joint  5 

5.00625  x  10“3  deg/bit 

4.69177  x  10-3  deg/bit 

i-30 
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Table  1-5.  Specifications  (Cont) 


ITEM 

SPECIFICATIONS 

PUMA  560 

Wrist  -  Joint  4 

4.7361  x  10~3  deg/bit 

Wrist  -  Joint  5 

5.00625  x  10“3  deg/bit 

Wrist  -  Joint  6 

4.69177  x  10“3  deg/bit 

Encoder  Index  Resolution 
(One  Motor  Revolution) 

PUMA  550/560 

Waist  -  Joint  1 

5.7557  deg 

Shoulder  -  Joint  2 

3.3392  deg 

Elbow  -  Joint  3 

6.7098  deg 

PUMA  550 

Wrist  -  Joint  4 

5. 0062  deg 

Wrist  -  Joint  5 

4.6918  deg 

PUMA  560 

Wrist  -  Joint  4 

4.7358  deg 

Wrist  -  Joint  5 

5.0062  deg 

Wrist  -  Joint  6 

4.6918  deg 

Velocity  at  SPEED  100 

PUMA  550/560 

Waist  -  Joint  1 

82.1  deg/sec 

Shoulder  -  Joint  2 

53.5  deg/sec 

Elbow  -  Joint  3 

122.1  deg/sec 

PUMA  550 

Wrist  -  Joint  4 

241.4  deg/sec 

Wrist  -  Joint  5 

227.8  deg/sec 

PUMA  560 

Wrist  -  Joint  4 

227.8  deg/sec 

Wrist  -  Joint  5 

241.4  deg/sec 

Wrist  -  Joint  6 

227.8  deg/sec 
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Table  1-5-  Specifications  (Cont) 


ITEM 

SPECIFICATIONS 

Maximum  Linear  Velocity 
at  Speed  100 

468  mm/sec 

Maximum  Cartesian  and 
Joint  Acceleration 
(=  Deceleration)  Time. 

Rest  to  Maximum  Velocity 

Cartesian 

0.112  sec 

Waist  -  Joint  1 

0.112  sec 

Shoulder  -  Joint  2 

0.112  sec 

Elbow  -  Joint  3 

0.056  sec 

Wrist  -  Joint  4 

0. 056  sec 

Wrist  -  Joint  5 

0.056  sec 

Wrist  -  Joint  6 

0.056  sec 

Maximum  Envelope  Error 
(Joint  Excursion  to 

Fatal  Error) 

PUMA  550/560 

Waist  -  Joint  1 

11.51  deg 

Shoulder  -  Joint  2 

8.34  deg 

Elbow  -  Joint  3 

13.41  deg 

PUMA  550 

Wrist  -  Joint  4 

10.01  deg 

Wrist  -  Joint  5 

9.38  deg 

PUMA  560 

Wrist  -  Joint  4 

9.47  deg 

Wrist  -  Joint  5 

10.01  deg 

Wrist  -  Joint  6 

9.38  deg 

Controller 

Dimensions 

0.48  m  (19  inches)  W  x  0.311  m 
(12.25  in.)  H  x  0.050  m  (19.7  in.)  D 

Weight 

356  N  (80  lb) 

1-32 
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Part  2:  Description  of  MIC  Vision  System  (18) 


Model  VS-100 
MACHINE  VISION  SYSTEM 


For  Inspection,  Material  Handling,  and  Assembly 

Simplified  Light  Pen  Control 

Easy  Programming  by  Showing 

Versatile  Camera  Inputs 

Programmable  Output  Control 

Powerful  Computing  Capability 

& 

i 

Machine  Intelligence  Corporation 
1120  San  Antonio  Road 
Palo  Alto.  California  94303 
<415)968-4008 


Model  VS- 


Functional  Block  Diagram 


Theory  of  Operation 

The  VS- 100  consists  of  one  or  more 
cameras  (any  of  four  types),  an  image  proces¬ 
sing  unit,  and  a  Digital  Equipment  LSI— 1 1 
microcomputer.  It  accepts  images  from  up  to 
two  separate  cameras  (A),  which  may  be  oper¬ 
ated  with  a  strobe  lamp  triggered  under  VS-1 00 
control.  The  image  processing  unit  (B) 
transforms  camera  images  into  binary  (black 
and  white)  images  that  are  run-length  encoded 
for  data  compression  and  subsequent  proces¬ 
sing.  Efficient  algorithms  operating  in  the  LSI-1 1 
(C)  perform  a  complete  connectivity  analysis  of 
the  encoded  images,  building  data  structures 
that  represent  essential  features  of  each  con¬ 
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tiguous  region.  Up  to  13  distinguishing  features 
such  as  area,  perimeter,  center  of  gravity, 
number  of  holes,  and  minimum  and  maximum 
radii  can  be  extracted  for  each  region. 

Object  recognition  is  performed  using  a 
nearest  neighbor  classifier  operating  on  a  user- 
selectable  subset  of  the  features.  Precise 
numerical  measurements  are  computed  to  in¬ 
dicate  the  degree  of  confidence  in  the  system's 
recognition  of  the  object.  If  the  degree  of  matr 
of  the  selected  features  is  above  a  user- 
adjustable  threshold,  the  object  is  rejected  as 
unknown  or  defective.  Once  an  object  is 
recognized,  its  position  and  orientation  are 
determined. 
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',;sion  System 

(  An  Eye  on  the  Future 

The  age  of  automated  vision  is  here.  In  the 
factory,  non-contact  visual  sensing  can  be  used 
for  inspection,  measurement  of  critical  dimen¬ 
sions.  parts  sorting,  programmable  part  presen¬ 
tation,  sensing  for  process  control,  and 
automated  assembly  For  a  modest  capital  in¬ 
vestment  (that  won't  increase  for  multi-shift 
operation),  machine  vision  provides: 

•  enhanced  productivity, 

•  improved  quality  control,  and 

•  greater  reliability 

Machine  Intelligence  Corporation  has 
utilized  the  results  of  over  a  decade  of 
government-sponsored  research  in  automated 
vision  and  engineered  the  VS-100  Vision 
System — a  rugged,  versatile,  complete  unit 
designed  for  industrial  applications 

The  VS-100  Vision  System 

The  VS-100  is  a  highly  advanced  system, 
with  both  a  broad  range  of  image  processing 
capabilities  and  ease  of  use.  It  recognizes  and 
inspects  images  of  complex  objects  against  a 
contrasting  background  in  real  time  The  objects 
can  be  anywhere  in  the  field  of  view,  in  any 
orientation,  even  on  a  moving  belt.  Thus,  a 
workpiece  randomly  oositioned  on  a  belt  or  a 
table  can  be  located  precisely,  so  that  it  can  be 
inspected,  or  acquired  and  manipulated  by  an 
industrial  robot  or  other  positioning  device. 

Ease  of  Use 

A  most  striking  feature  of  the  VS-100  is  its 
numan  engineering.  The  VS-100  can  be  trained 
to  analyze  new  objects  sirnDly  by  showing  them 
to  the  system  Interactions  with  the  system  are 
menu-driven,  using  light-pen  input:  for  most 
applications,  users  never  need  to  touch  a 
Keyboard!  Easy-to-use  menus  allow  selection  of 
the  threshold  level,  relevant  features,  and  other 
important  parameters  for  specific  applications. 
System  control  is  by  light  pen,  too  Calibration, 
training-by-showing,  and  storing  and  loading  of 
prototype  data  can  easily  be  done,  with 
minimum  training. 

A  Complete  Development  Tool 

Because  of  its  generality  and  simplicity  of 
use,  the  VS-100  is  ideal  for  developing  and 
assessing  applications  of  visual  sensing.  The 
VS-100  is  a  complete  system  including  display, 
light  pen,  storage  device  fcr  prototype  informa¬ 
tion  and  a  camera  of  your  choice,  as  well  as 
the  visual  processing  unit.  Vet  its  modular, 
expandable  design  permits  easy  system  integra¬ 
tion 


The  VS-100  accepts  on-off  signals  from 
external  devices,  and  can  in  turn  send  on-off 
signals  to  other  devices — up  to  sixteen  in  all. 
Three  serial  ports  are  available  for  commun¬ 
ication  with  other  computers — large  hosts  for 
system  development  or  microcomputers  for 
controlling  gates,  X-Y  tables,  industrial  robots, 
or  other  manipulators.  Special  interfaces  to 
other  PDP-l  1  computers  and  several  robots  are 
available.  The  LSI-1 1  bus  allows  efficient 
interaction  with  a  wide  range  of  compatible 
devices  (such  as  A/D  and  D/A  converters  and 
other  LSI-1 1  modules)  which  you  can  add  for 
your  own  purposes 

A  Cost-Effective  Application  Device 

The  many  sophisticated  features  of  the 
VS-100  have  been  engineered  for  fast,  consis¬ 
tent  operation  in  a  production  environment. 
Where  the  system  can  be  used  without  a  large 
investment  in  ancillary  equipment,  the  cost  of 
the  unit  can  be  justified  for  a  one-shift  opera¬ 
tion.  In  most  instances,  a  complete  system  in¬ 
cluding  the  VS-100  and  some  computer- 
controlled  manipulation  provides  adequate 
return  on  investment  for  a  two-shift  operation. 
The  VS-100  provides  high  reliability:  it  operates 
at  full  accuracy  around  the  clock,  every  day  of 
the  week.  Battery  power  backup  is  an  available 
option.  Application-specific  functions  can  be 
programmed  by  our  staff. 

What  the  VS-100  Can  Do  for  You 

Examples  of  applications  in  which  the 
VS-100  can  improve  product  quality  and  reduce 
costs  include: 

Inspection- 

Integrity  and  completeness 
Shape  and  size  defects 
Flash 

Number,  size,  and  positioning  of  holes 
Cosmetic  stains 
Cracks  and  burrs 

Measurement  of  critical  dimensions 

Sensor-controlled  acquisition  and 

manipulation- 
Workpiece  sorting 
Workpieces  randomly  positioned  on 
conveyors 

Manufacturing  processes  requiring  visual 
feedback 

Fastening  operations 
In-prccess  inspection 
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SPECIFICATIONS 
Model  VS-100  Vision  System 

General  Features 

•  Human  engineered  to  provide  interactive  menu-driven  selection  of  threshold  level,  relevant  features  and  other  important 

parameters  for  specific  applications,  using  light-pen  input. 

•  Training-by-showing  for  simplified  programming. 

•  Binary  threshold  settable  by  either  light  pen  (interactively)  or  by  program  control. 

•  Settable  black  or  white  background. 

•  Fast  run-length  data  compression  hardware. 

•  Single  pixel  noise  rejection-optionatly  settable. 

•  Software  windowing  of  image  area  by  light  pen  control. 

•  Menu<)riven  connectivity  analysis  software. 

•  Optionally  reprocessing  of  selected  image  data. 

•  Menu-driven  feature  selection  and  calibration. 

•  DEC  LSI-1 1/2  computer  with  EIS/FIS  and  64K  Byte  RAM. 

•  Cassette  recorder  for  program  loading  and  prototype  storage. 

•  2  strobe  lamp  triggers  for  "freezing"  moving  objects. 

•  2  quad-slots  or  4-dual-slots  available  on  LSI-1 1  backplane  for  customized  applications,  e  g.,  A/D,  D/A,  TTL  interfaces. 

•  Processing  time  for  many  applications  within  a  fraction  of  a  second. 

Input/Output 

•  Can  accommodate  up  to  2  cameras,  singly  or  in  combination  from  the  following  types: 

a.  GE  TN2200,  128  x  128  solid  state  array  (standard) 

b.  GE  TN2500.  240  x  240  solid  state  array  (optional) 

c  Reticon  LC600C256-1 .  256  x  i  solid  state  array  (optional) 
d.  Standard  (RS-170)  vidicon  with  external  sync  (optional) 

•  Light  pen  input 

•  4-port  serial  RS-232C.  RS-422  or  RS-423  interface  (0LV11-J) 

•  1 6— bit  parallel  TTL  interface  (DRV1 1) 

•  2  frame  buffers  for  up  to  256  x  256  pixel  arrays  (binary) 

•  1  display  buffer  (256  x  256)  for  graphics  and  text  overlays 

•  1  12"  TV  Display  Monitor  for  displaying  binary  image  data,  processed  image  data,  and  analog  images 

Communications  Interlace 

•  Standard  RS-232C.  RS-422  or  RS-423  serial  communication  ports  for  host  and/or  control  computers. 

•  Data  rate  jumper-selectable  from  150  to  38,400  baud. 

Manipulator  Interface 

•  Bi-directional  1 6— bit  parallel  interface  may  be  used  for  manipulator  control  applications, — communication  protocols 

are  available  for  several  major  industrial  robots. 

Mechanical 

•  Cabinet  enclosure.  Can  also  be  installed  in  a  standard  19"  rack 

•  Rack-mount  dimensions.  5'/*"  high:  17"  wide:  24"  deep 

•  Weight  approximately  50  lbs. 

Electrical 

•  Power:  105-125  VAC.  60  HZ.  300  watts  maximum 

•  Optional:  Battery  stand-by  supply,  will  maintain  program  and  data  storage  for  up  to  4  hours. 

Warranty 

•  The  vs-100  Machine  Vision  System  is  guaranteed  to  be  free  of  manufacturing  parts  and  labor  defects  for  a  period  of  90 

days  after  delivery. 

For  further  information  or  applications  assistance,  please  write  or  call: 
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SECTION  I 


INTRODUCTION 

Univision  is  a  camera-based  processing  system  which  enables  VAL,  the 
UNIMATE  control  system,  to  locate  and  identify  objects  within  a  given 
work  area.  The  Univision  system  is  composed  of  three  subsystems, 
namely: 

(1)  Area  array  camera  sensors  (maximum  of  two) 

(2)  Vision  processor,  including: 

(a)  Machine  Intelligence  Corporation  (MIC)  VS-100  Processor 

(b)  Light  pen  and  monitor  for  user  interaction  with  the 
vision  system 

(3)  Univision  Interface  Kit  enabling  VAL,  appended  with  vision 
commands,  to  coummunicate  with  the  vision  processor.  The  Kit 
includes : 

(a)  Interprocessor  Communications  Interface  Card 

(b)  Interprocessor  Communications  Cables 

(c)  Vision  software  and  diagnostics  (down  loadable  from  VAL's 
f loopy  disk) 

A  block  diagram  of  the  hardware  system  is  shown  in  Figure  1.1. 

The  Univision  svstem  locates  and  cateqorizes  stationarv,  objects  within 
its  field  of  view  on  the  basis  of  features  extracted  from  each  object's 
silhouette  -  silhouettes  generated  from  a  spatially  ouantized  image 
obtained  from  a  single,  fixed,  overhead  view.  The  control  system,  VAL, 
transforms  the  object  location/orientation  information  into  robot 
coordinates  so  that  it  can  identify  an  appropriate  user-defined, 
object-dependent  grasp  point(s),  and  then  pick  up  and  move  the  object  as 
desired . 

The  key  element  in  Univision  is  the  VS-100  processor  -  a  soohisticated 
image  processing  and  pattern  recognition  system.  Most  of  its 
sophistication  is  hidden  from  the  user  by  its  easy-to-use,  high-level, 
menu-driven  architecture.  However,  effective  application  of  the 
Univision  system  will  require,  in  most  cases,  that  the  user  have  a  basic 
understanding  of  the  "behind-the-scenes"  operation  of  this  device.  The 
next  section  will  discuss  the  operation  of  the  Univision/VAL  control 
system  and  will  focus,  in  particular,  on  the  VS-100  processing. 

1.1  FUNCTIONAL  OPERATION  OF  THE  UNIVISION/PUMA  CONTROL  SYSTEM 

Figure  1.2  shows  a  functional  block  diaqram  of  the  Univision/PUMA 
control  system.  The  diagram  illustrates  the  processing  functions 
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necessary  to  locate,  recognize,  and  pick  up  parts.  The  basic  functions 
are: 


(1) 

Image  Formation 

(2) 

Boundary  Detection 

(3) 

Location  and  Orientation  Computation 

(4) 

Extraction  of  Orientation-Independent 

Features 

(5) 

Object  Classification  (Recognition) 

(«) 

Inter-Processor  Communication 

(7) 

Transformation  of  Object  Location/Orientation  from  the  Vision 
Frame  to  the  Robot  Frame 

(8) 

Control 

We  will  now  describe,  in  detail,  the 

Univision/PUMA  during  a  typical  vision-aided 

internal  operation  of 

task.  We'll  consider  the 

task  of  picking  up  a  wrench,  located  and  oriented  arbitrarily  within 
Univision's  field  of  view,  and  placing  it  in  a  bin.  The  steos  required 
to  complete  this  task  are  as  follows: 

1.  VAL  initiates  the  task  by  sending  a  message  to  the  vision 
processor  asking  it  to  freeze  the  current  video  frame  (picture) 
for  further  processing.  One  frame  of  video,  consistina  of  an 
array  of  N1  x  N2  analog  voltages  fqray  scale) ,  each  proportional 
to  the  image  intensitv  on  a  particular  photo  element  (pixel)  in 
the  solid  state  array,  is  sent  to  the  vision  processor.  The  N2 
"column  voltages"  are  read  out  continuously  in  a  raster-type  scan 
for  each  of  the  Nl  rows  of  the  sensor  array  grid. 

2.  The  vision  system  thresholds  the  video,  on-the-fly,  thus  creating 
a  binary  image  or  silhouette.  The  thresholding  operation  defines 
each  object's  outline  or  boundary.  To  further  simplifv  processing, 
only  the  position  of  threshold  crossinas  (positive  and/or  negative) 
and  the  number  of  pixels,  or  run  length,  in  between  each  crossing 
are  stored  in  image  memory  for  each  raster  scan  line.  This  coding 
process  is  called  run  length  encoding  and  results  in  near  minimal 
storage  of  the  information  in  the  binary  picture. 

3.  The  vision  system  performs  connectivitv  analvsis  which  combines 
all  run  lengths  which  are  part  of  an  object's  silhouette.  This 
analysis  also  detects  the  presence  of  holes.  Anv  objects  whose 
silhouettes  merge  are  subsequently  treated  as  a  single  object 
after  this  stepl 

4.  In  parallel  with  "3.",  the  vision  processor  sums  the  centroids  of 
all  of  the  run  lengths  for  each  silhouette  to  obtain  its  centroid 
location  in  camera  coordinates.  Orientation,  using  anv  of  a  varitc 


100 


of  orientation  algorithms,  is  also  computed. 

Feature  extraction,  the  process  of  computing  a  set  of  quantities 
that  can  be  used  in  object  (nattern)  classification,  is  then 
performed  by  the  vision  Drocessor.  In  Univision,  only  orientation/ 
position-independent  features  are  computed  because  a  part  must  be 
recognized  regardless  of  its  placement  in  the  field  of  view.  Tvpical 
features  computed  are  area,  compaction  (perimeter  *  *  2/area),  second 
moments,  number  of  holes,  etc.  The  VS-100  allows  the  user  to  choose 
these  features,  among  a  set  of  13  and  aids  the  user  in  selecting 
those  features  which  are  best  for  a  given  application. 


Object  classification  is  then  oerformed  by  statistically  comparing 
the  computed  features  of  the  silhouette  with  those  of  silhouettes 
generated  by  a  set  of  prototype  objects.  Typically,  the  prototypes 
are  the  set  of  objects  that  one  would  expect  to  "see"  at  the 
workstation.  The  user  trains  Univision  to  recoanize  these  parts. 
During  the  training  Drocess,  the  vision  system  establishes  the  mean 
and  variability  of  the  features  of  the  Drototvoe  silhouette  over  a 
variety  of  positions  and  orientations  within  the  field  of  view  and 
over  the  possible  stable  states  of  the  prototype  object. 

Silhouettes  are  classified  as  being  a  "member"  of  the  Drototype 
group  if  the  statistical  "distance"  between  the  features  of  the 
prototype  and  the  silhouette  is  shortest.  This  technique  is  called 
"nearest-neighbor"  classification.  The  following  example  will 
explain  the  process  and  ooint  out  the  imoortance  of  prototype 
training . 

EXAMPLE:  We  will  suppose  that  (1)  our  prototvpes  are  limited  to 

two  objects  -  an  adjustable  wrench  and  a  fixed  head  wrench;  and, 

(2)  the  feature  that  most  easilv  classifies  them  is  the  compaction 
index,  c  *  Derimeter  *  *  2/area.  During  training,  we  "show"  the 
vision  system  the  two  objects  in  a  variety  of  possible  locations, 
and  orientations.  In  the  case  of  the  adjustable  wrench,  we  also 
train  on  the  expected  range  of  wrench  openings.  This  training 
procedure  defines  the  mean  and  variability  of  the  features  used 
during  classification.  The  spread  (probability)  functions  of  c 
for  the  two  objects  after  training  are  illustrated  in  Figure  1.3. 

As  shown,  the  adjustable  wrench  exhibits  higher  feature  variability 
because  its  perimeter  can  vary  signif icantlv,  as  a  function  of 
opening,  without  significant  changes  in  area. 

The  statistical  distance  from  a  measured  feature,  c,  to  the  average 
feature  of  a  prototype  c  is  measured  in  terms  of  the  number  of 
standard  deviations  d  between  c  and  c.  If  defines  the 
prototype  variability  established  during  orototvping,  then  the 
distance,  d,  is 
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d  is  in  a  sense  a  probability  measure  -  as  d  increases,  the 
probability  that  the  object  matches  the  prototyoe  decreases. 

For  example,  if  d  >  3,  and  the  soread  function  of  the  prototype 
feature  is  normally  (Gaussian)  distributed,  there  is  less  than  a 
1%  chance  that  the  object  is  a  member  of  the  prototype  classification 

The  nearest  neighbor  classification  procedure  computes  the  "distance" 
between  the  object  feature(s)  and  each  prototype's  feature(s).  The 
object  is  classified  as  a  member  of  the  prototype  qroup  if 

(1)  its  feature  "distance",  d*,  to  the  prototype  is  shortest;  and, 

(2)  d*  <  dmax  where  dmax  establishes  a  statistical  confidence  limit 
beyond  which  the  object  cannot  be  classified  -  the  probability  that 
the  object  is  a  member  of  any  prototype  group  is  too  small. 

If  we  refer  to  Figure  1.3  we  see  then  that  van  object  with  compaction 
index,  cl,  would  be  classified  as  an  adjustable  wrench.  Note, 
however,  that  the  numerical  value  of  Cl  is  actually  closer  to  c“ 
measured  for  a  fixed  head  wrench.  An  object  with  feature  c2  cannot 

be  classified  since  it  clearly  lies  outside  any  reasonable 
confidence  interval  of  beinq  a  member  of  either  the  wrench 
classifications. 

As  can  be  seen,  prototype  training  is  key  to  reliable  object 
classification.  Care  must  be  taken  by  the  user  to  ensure  that  the 
mean  and  variability  of  silhouette  features  measured  during  trainina 
adeauatelv  reflects  that  which  will  be  encountered  in  practice. 


The  nearest-neiqhbor  classification  algorithm  is  done  "behind-the- 
scenes"  in  the  VS-100  and  normally  requires  no  user  intervention 
except  during  prototype  training.  In  this  regard,  the  VS-100 
system  helps  the  user  to  train  prototypes  and  allows  him  to 
evaluate,  improve,  and  predict  classification  performance  before 
he  puts  the  system  on-line. 

7.  Upon  receiving  a  completion  message  on  the  original  picture  command, 
VAL  asks  the  vision  processor  for  the  location  of  any  objects  that 
have  been  classified  as  a  wrench. 


8.  The  vision  system  reports  the  location  and  orientation  of  a 
wrench  (if  th<ere  is  one)  in  its  field  of  view  by  sending  an 
appropriate  message  to  VAL.  Note  that  during  the  time  the  vision 
system  is  locating  and  identifying  objects,  VAL  Is  free  to  complete 
other  tasks.  This  overlapping  mode  of  operation  minimizes  the 
effect  of  vision  system  processing  on  robot  cvcle  times. 

9.  In  the  final  step  of  the  task,  VAL: 

(a)  Transforms  the  location  and  orientation  of  the  wrench  from 
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vision  to  robot  coordinates  using  parameters  defined  during 
calibration. 

(b)  Determines  the  location  and  orientation  of  a  user  defined 
grip  point.  This  feature  allows  a  wide  range  of  objects  to 
be  picked  up  in  a  reliable  fashion. 

(c)  Checks  with  the  VS-100  to  see  whether  the  robot  gripper  will 
have  clearance  to  pick  up  the  part  at  the  grip  location. 

(d)  Positions  and  orients  the  gripper. 

(e)  Picks  up  the  wrench  and  places  it  in  the  bin. 

Even  a  simple  task  like  that  described  above  reauires  significant  vision 
processing  capability.  VAL  has  been  programmed  so  that  only  a  few, 
high-level,  vision  commands  are  needed  to  access  information  fiom  the  VQtt) 
processor  and  to  control  its  complex  image  processing  and  pattern 
recognition  ooerations.  This  capability  frees  the  VAL  user  to 
concentrate  more  on  the  important  control  aspects  of  a  vision-aided 
task. 


1.2  OVERVIEW  OF  THE  MANUAL 

The  manual  covers  the  theory,  operation,  maintenance,  installation,  and 
user  application  of  Univision.  Chapter  2  is  a  copy  of  the  VS-100 
processor  reference  manual  which  describes  the  setup  and  operation  of 
the  vision  system  and  its  peripherals.  Chapter  3  is  a  supplement  to 
"Users  Guide  to  VAL"  that  describes  the  new  VAL  commands  used  for 
interrogation  and  control  of  the  vision  system.  Chapter  4  is  a  user's 
guide  to  Univision  which  covers  (1)  practical  considerations  in  setting 
up  and  using  a  Univision  system;  (2)  setup  of  the  camera  system;  (3) 
installation  of  the  Univision  interface  Kit  and  start-up  procedures;  (4) 
vision  and  vis ion- to- robot  calibration;  (5)  prototype  training;  (6) 
computing  object  to  robot  transformations;  and,  (7)  application 
programs.  Chapter  5  contains  an  operating  manual  for  the  cameras (either 
TN-2500  or  MIC  22)  used  in  the  application.  Chapters  6,  7,  8  cover  the 
spare  parts  list,  mechanical  drawings  and  electrical  drawings 
respectively. 

Three  sections  are  included  in  the  Appendix.  Appendix  A  is  the 
diagnostics  manual  for  the  VS-100  describing  use  of  the  diagnostic 
software  and  other  debugging  procedures  needed  to  isolate  failures  in 
the  VS-100  system.  Appendix  B  describes  in  detail  the  hardware  and 
software  protocols  used  in  the  interprocessor  communications  interface. 
Appendix  C  is  a  copy  of  an  applications  note  covering  lens  selection  for 
the  TN-2500  and  TN-2200  (same  sensor  as  the  MIC-22)  General  Electric 
camera  sensors. 

1.3  FUNCTIONAL  SPECIFICATION 

Parts  Recognition: 
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Maximum  number  of  trained  orototypes  resident  in 
the  system  at  a  given  time 


9 


Maximum  number  of  objects  that  can  be  recognized 
in  the  field  of  view  12 

Reliable  parts  recognition  requires  adequate  image  contrast  and 
requires  that  the  oart  silhouettes  do  not  merge. 

Vision  Sensor  Resolution: 

MIC  22  1  Part  in  128  of  Camera  Field  of  View 

G.E.  TN-2500  1  Part  in  244  of  Width  of  Field 

1  Part  in  248  of  Height  of  Field 
Location  and  Orientation  Measurement  Accuracy 


(Iqnoring  effects  of  parallax,  and  robot  vision  calibration  errors) 

Location  Equal  to  or  Better  Than  Vision  Sensor  Resolution 

Orientation  Dependent  on  Object  and  on  the  Orientation 

Feature  Selected 


Environmental : 

Camera  w/o  Enclosure 
Ambient  Temp 
Humidity 

Vision  Processor 
Ambient  Temo 
Humidity 


o 

0-50  C 

90%  Max  Non-Condensing 
o 

0-46  C 

90%  Max  Non-Condensing 
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Part  3:  Interface  Software  Descriptions  (19) 
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This  document  attempts  to  describe  all  of  the  improvements  that  Univiaion 
VAL-I I  has  compared  to  Univisionl  VAL-I. 

Additions: 

0. blink. blob  -  This  routine  will  allow  the  user  from  the  VAL  keyboard 
to  outline  the  selected  blob  with  either  a  white  or  a 
black  border.  This  routine  can  be  very  helpful  in 
determining  which  blob  the  vision  system  is  refering  to. 

U.Boot  -  Allows  the  VAL-IZ  user  to  restart  the  vision  system  from  the  VAL-I 
keyboard.  It  is  a  destructive  restart  in  that  it  will  delete  all 
prototypes.  However,  it  can  save  the  user  from  having  to  reload  t 
vision  system  software  from  tape. 

U. Delete. Blob  -  This  routine  will  allow  the  user  to  delete  blobs  from  the 
blob  descriptor  area.  This  will  save  vision  system  memory 
allowing  more  trainings  per  prototype. 

U. Erase  -  This  routine  will  erase  the  text  overlay,  this  will  allow  the 
user  to  see  a  clear  unobstructed  digitized  picture. 

0. Recognize  -  This  routine  will  compare  all  of  the  known  prototypes  to 
to  the  specified  blob  and  return  the  prototype  id  number 
that  best  fits  the  blob.  The  routine  will  also  return 
the  difference  from  the  best  fit,  and  the  difference  to 
the  second  best  fit. 

U. Raster .Line  -  Allows  the  user  to  examine  each  line  on  the  vision  system 
to  determine  which  pixels  are  on  or  off. 


Overall  Improvements- 

1.  By  selecting  the  fast  option  VAL-II  can  process  other  information 
while  the  vision  system  is  processing  data.  Upon  completion  VAL-i; 
can  readdress  the  vision  system  to  get  the  processed  data. 

2.  Overall  the  user  has  more  options  when  analyzing  the  vision  picture 
Changes- 

Pindheap-  Now  is  U. WHERE. ARE. YOU. PROTO.  Information  is  contained  in 
array  element  (0] . 

Train-  Handled  by  U. TRAIN. BLOB.  User  can  now  use  teach  pendant  to 
train  prototypes. 

VS TORE,  VL0AD-  Functions  handled  by  U. SAVER,  however,  U. SAVER  does  not 
preserve  switch  settings,  and  orientation  features. 

U. SAVER  uploads  the  information  into  an  array  then  the 
user  must  store  the  array  to  disk  as  a  series  of  real 
values.  U. SAVER  will  allow  the  user  to  delete  prototype 

Not  Xmplemented- 

Grip  functions. 
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Routine:  U.Boot 


Description: 

This  routine  will  allow  the  user  to  boot  the  NIC  vision  system.  CAUTION: 
during  a  reboot  all  prototype  information  will  be  erased.  This  routine  has 
been  provided  for  use  when  the  vision  system  hangs. 

Usage : 


exec  boot 


Created:  Sept  8,  198S 
Revision:  1.0 


Routine:  0. Blink. Blob 
General  Description: 

This  diagnostic  routine  will  aid  in  locating  blobs  by  outlining  them  with  a 
white  or  a  black  border. 

Variables  Description: 


u.blobnum  -  This  parameter  must  be  set  by  the  user  before  callinq  this 
routine. 

The  legal  values  range  from  one  to  the  number  of  blobs  that 
are  currently  on  the  display  screen. 

u. color  -  This  parameter  must  be  set  by  the  user  before  calling  this 

routine. 

The  variable  has  two  legal  values,  zero  and  negative  one. 

If  the  variable  is  set  equal  to  negative  one,  the  outline  is 
drawn  in  bright  white. 

If  the  variable  is  set  eoual  to  zero,  the  outline  is  drawn  in 
black. 

s. error  -  This  variable  indicates,  in  a  logical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicate 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s. system. error.  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 

s. messaged  -  Reserved  for  system  use. 

s.numbytes  -  Reserved  for  system  use. 

s.nmpwords  -  Reserved  for  system  use. 

s .system .error  -  This  variable  contains  the  error  code  that  is  returned  from 
the  vision  system.  It  should  be  referenced  when  the  variabj 
s. error  indicates  that  a  problem  occured  with  the  routine. 

If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 

Subroutine  Usage: 

U. Blink. blob  directly  calls: 

S. transfer .message 
S.get .message 

Usage  Example: 

.Program  Test 

1, . . . . . . . 

2: •  This  program  will  flash  a  blobs'  border  white  and  black  until 
3;*  the  user  strikes  the  <REC>  key  on  the  teach  pendant 
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4  PROMPT  "PLEASE  ENTER  THE  BLOB  NUMBER  TO  BLINK  — >  ",S.BLOBNUM 

5  TYPE  "PRESS  <REC>  ON  THE  PENDANT  TO  STOP" 

6  DO 

7  U. COLOR— 1  ;SET  COLOR  TO  WHITE 

8  CALL  U. BLINK. BLOB;  TELL  VISION  TO  MAKE  BORDER  WHITE 

9  IF  S. ERROR  THEN 

10  TYPE  "ERROR:  HALTING"  ;  STOP  IF  AN  ERROR 

11  HALT 

12  END 

13  FOR  INDEX  «  1  TO  500 

14  END 

15  U.COLOP-O  ;SET  COLOR  TO  BLACK 

16  CALL  U. BLINK. BLOB  ;TELL  VISION  TO  MAKE  BORDER  BLACK 

17  IF  S. ERROR  THEN 

18  TYPE  "ERROR:  HALTING"  .-STOP  IF  ERROR 

19  HALT 

20  END 

21  UNTIL  PENDANT ( 1 )  BAND  1  «*»  1 
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Routine:  0. Change. camera 
General  Description: 

This  routine  will  allow  the  user  to  select  the  camera  that  will  be  used. 
Variables  Description: 


u. camera  -  This  variable  must  be  set  by  the  user  before  this 

routine  is  called.  It  must  contain  a  number  between 
one  and  four,  (inclusive)  The  number  that  this  variable 
contains  will  be  the  new  camera  number. 


s .error 


s.message[] 
s .numbytes 
s  .numword  s 


This  variable  indicates,  in  a  logical  fashion,  the  result  of 
the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicate 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s .system .error .  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 

Reserved  for  system  use. 

Reserved  for  system  use. 

Reserved  for  svstem  use. 


s .system .error  -  This  variable  contains  the  error  code  that  is  returned  from 
the  vision  system.  It  should  be  referenced  when  the  variabl 
s. error  indicates  that  a  problem  occured  with  the  routine. 


If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 


Subroutine  Usage: 

U. Change. camera  directly  calls: 

s .  transfer  .messaqe 
s .get .message 

Usage  Example: 

.proqram  usage 

1  U. camera-1 

2  CALL  U. change. camera 

3  CALL  U. take. picture 

4  FOR  de.lay-1  TO  500 

5  END 

6  U. camera-2 

7  CALL  U. change. camera 

8  CALL  U. take .picture 

9  FOR  delay-1  TO  500 

10  END 
exec  usage, -1 
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Routine:  0. Delete. Blob 


General  Description: 

This  routine  will  remove  a  blobs*  features  from  the  MIC  vision  system. 
(It  does  not  delete  a  prototype.) 


Variables  Description: 


u.delete.blob.num  -  This  variable  must  be  set  by  the  user  before  calling  this 

routine. 


s. error  -  This  variable  indicates,  in  a  logical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicates 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s .system .error .  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 


s  .messaged 
s .  numbytes 
s .  numword  s 


Reserved  for  system  use. 
Reserved  for  system  use. 
Reserved  for  system  use. 


s .system .error  -  This  variable  contains  the  error  code  that  is  returned  from 

the  vision  system.  It  should  be  referenced  when  the  variable 
s. error  indicates  that  a  problem  occured  with  the  routine. 


If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 


Usage  Example: 

To  remove  blob  descriptor  2  from  the  vision  system 
.DO  U. DELETE. BLOB. NUM-2 
.EXEC  U. DELETE. BLOB 
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Routine:  U. Erase 
General  Description 

This  routine  will  erase  the  any  text  or  graphic  overlays  on  the  screen.  The 
routine  does  not  erase  the  current  picture  in  the  buffer. 

Variables  Description: 


s. error  -  This  variable  indicates,  in  a  loqical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicates 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s .system .error .  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 


s .message [ ] 
s .numbytes 
S  .numword  s 


Reserved  for  system  use. 
Reserved  for  system  use. 
Reserved  for  system  use. 


s .system .error  -  This  variable  contains  the  error  code  that  is  returned  from 

the  vision  system.  It  should  be  referenced  when  the  variable 
s. error  indicates  that  a  problem  occured  with  the  routine. 


If  s. error  is  FALSE,  this  variable  will  be  egual  to  2ero. 


Subroutine  Usage: 


Erase  directly  calls: 


S . transfer .message 
S.get .message 


Usage  Example: 


•Program  Demo 

1  Call  U. erase;  Erase  vision  screen  to  make  look  good. 
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Routine:  U.Initvis ion 
General  Description: 

This  routine  will  initialize  the  communication  software.  It  will  also  set 
default  values  for  other  the  other  subroutines.  This  routine  must  be  called 
before  any  other  communication  is  attempted. 

Variables  Description: 


Setup  Section: 

U.blobnum  -  Initialized  value  is  one.  (Set  primary  blob) 

U. color  -  Initialized  value  is  neqative  one.  (Color*  white.) 

U. expected .shapes  -  Initialized  value  is  nine. 

U.fast  -  Initialized  value  is  zero.  (Slow  Picture) 

0. new. picture  -  Initialized  value  is  zero.  (No  picture) 
U.numblob. id  -  Initialized  value  is  negative  one.  (Identify  all) 
0. picture. fast  -  Initialized  value  is  zero.  (Slow  Picture) 

U. camera  -  Initialized  value  is  one. 

Variables  Section: 


s. error  -  This  variable  indicates,  in  a  logical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicates 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s .system. error.  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 


s  .exp 
s .fraction 
s .message [ ] 
s  .numbytes 
s .  numword  s 


Reserved  for  system  use. 
Reserved  for  system  use. 
Reserved  for  system  use. 
Reserved  for  system  use. 
Reserved  for  system  use. 


s .system .error 


This  variable  contains 
the  vision  system.  It 
s. error  indicates  that 


the  error  code  that  is  returned  from 
should  be  referenced  when  the  variable 
a  problem  occured  with  the  routine. 


If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 


Subroutine  Usage: 


Initvision  directly  calls: 

S. transfer .message 
S.get .message 


Usage  Example: 


EXEC  U. INITVISION 


Routine:  U. Recognize 
General  Description: 

This  routine  will  compare  all  known  prototypes  to  the  specified  blob  number  and 
return  the  prototype  identification  number  that  corresponds  to  that  blob.  The 
routine  will  also  return  a  two  match  numbers  that  indicate  how  close  the  blob 
is  to  first  choice,  and  how  close  of  a  match  it  is  to  a  second  choice. 

Variables  Description: 

s.diffl  -  This  variable  contains  the  difference  from  the  specified  blob 

to  the  best  matching  prototype. 

s.diff2  -  This  variable  contains  the  difference  from  the  specified  blob 

to  the  next  best  prototype. 

CAUTION:  If  the  values  contained  in  s.diffl  and  s.diff2  are  very  close,  this 

indicates  that  the  vision  system  may  not  always  correctly 
differentiate  between  the  two  prototypes.  To  correct  this  reauires 
further  training  of  both  of  the  prototypes. 

s. error  -  This  variable  indicates,  in  a  logical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicates 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s  .system. error.  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 

s. message!]  -  Reserved  for  system  use. 

U.numblob  -  This  variable  must  be  set  by  the  user  before  calling  this 

routine.  It  contains  the  blob  number  that  you  want  compared 
to  all  of  the  known  prototypes. 

s.numbytes  -  Reserved  for  system  use. 

s.numwords  -  Reserved  for  system  use. 

s. proto. id  -  This  variable  contains  the  ID  number  of  the  prototype  that 

best  matches  the  specified  blob  number. 

s .system-. error  -  This  variable  contains  the  error  code  that  is  returned  from 

the  vision  system.  It  should  be  referenced  when  the  variable 
s. error  indicates  that  a  problem  occured  with  the  routine. 

If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 
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Routine:  U. Reprocess .picture 

General  Description: 

This  routine  will  instruct  the  vision  system  to  reanalyze  the  current  image. 

Variables  Description: 

s. blob. count  -  Upon  completion  of  the  routine  “Take .picture"  this  variable 
will  contain  a  count  of  the  number  of  blobs  that  are  in  the 
picture.  The  count  is  not  always  equal  to  the  number  of 
obiects  but  rather  is  the  number  of  connected  regions  in 
the  imaqe. 

s. error  -  This  variable  indicates,  in  a  logical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicates 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s .system. error.  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 

s. messaged  -  Reserved  for  system  use. 

s.numbytes  -  Reserved  for  system  use. 

s.numwords  -  Reserved  for  system  use. 

u.fast  -  This  parameter  must  be  set  by  the  user  before  calling  the 

routine . 

u.fast  has  two  legal  values;  zero  or  one. 

If  u.fast  is  set  equal  to  zero,  then  the  vision  system 
will  take  a  slow  picture.  Slow  means  that  the  routine  will 
wait  for  the  vision  system  to  completely  process  the  command 
before  proceeding. 

If  u.fast  is  set  equal  to  one,  then  the  vision 

system  will  take  a  fast  picture.  The  routine  will  finish 

before  vision  completes  all  processing.  Variable 

s. blob. count  will  be  equal  to  zero  regardless  of  the  number 

of  blobs  that  exsist  in  the  imaqe. 


s .system .error  -  This  variable  contains  the  error  code  that  is  returned  from 

the  vision  system.  It  should  be  referenced  when  the  variable 
s. error  indicates  that  a  problem  occured  with  the  routine. 

If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 


Subroutine  Usage: 

Reprocess .picture  directly  calls: 

S. transfer .message 
S. get. message 
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Routine:  0. Take. picture 
General  Description: 

This  routine  will  instruct  the  vision  system  to  take  a  picture  of  the  current 
field  of  view  and  process  the  image. 

Variables  Description: 

s. blob. count  -  Upon  completion  of  the  routine  "Take .picture"  this  variable 
will  contain  a  count  of  the  number  of  blobs  that  are  in  the 
picture.  The  count  is  not  always  eaual  to  the  number  of 
obiects  but  rather  is  the  number  of  connected  regions  in 
the  image. 

s. error  -  This  variable  indicates,  in  a  loaical  fashion,  the  result  of 

the  execution  of  the  routine.  If  it  returns  TRUE,  it  indicates 
that  an  error  was  detected  by  the  vision  system.  The  actual 
error  code  will  be  found  in  s .system. error.  If  it  returns  as 
FALSE,  the  routine  completed  with  out  an  error. 

s.messaget]  -  Reserved  for  system  use. 

s.numbytes  -  Reserved  for  system  use. 

s.numwords  -  Reserved  for  system  use. 

u. picture. fast  -  This  parameter  must  be  set  by  the  user  before  callinq  the 
routine. 

u. picture. fast  has  two  legal  values;  zero  or  one. 

If  u. picture. fast  is  set  eaual  to  zero,  then  the  vision  system 
will  take  a  slow  picture.  Slow  means  that  the  routine  will 
wait  for  the  vision  system  to  completely  process  the  command 
before  proceeding. 

If  u. picture. fast  is  set  equal  to  one,  then  the  vision 
system  will  take  a  fast  picture.  The  routine  will  finish 
before  vision  completes  all  processing.  Variable 
s. blob. count  will  be  eaual  to  zero  regardless  of  the  number 
of  blobs  that  exsist  in  the  image. 


s .system .error  -  This  variable  contains  the  error  code  that  is  returned  from 

the  vision  system.  It  should  be  referenced  when  the  variable 
s. error  indicates  that  a  problem  occured  with  the  routine. 

If  s. error  is  FALSE,  this  variable  will  be  equal  to  zero. 


ubroutine  Usage: 

U. Take. Picture  directly  calls: 

S. transfer  .message 
S. get .message 


sage  Example: 

Program  Test 
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U. picture. fast  *  0;  Set  slow  mode. 

Call  0. Take. picture;  Ask  the  vision  system  to  take  a  picture. 

If  S. error  then;  Examine  the  variable  to  see  if  a  problem  occured . 
Type  "Bad  Routine  Error  code  is  — >  "  ,/d ,s .system. error 
The  above  line  will  display  the  error  code  and  a  short  message. 

Else 

Type  "Good  Picture  Blob  Count  is  — >  " ,/d  ,s .blob. count 
The  above  will  happen  if  everything  worked. 

End 
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utine:  U. Where. Are. You. Pro to 
neral  Description: 

is  routine  will  analyze  an  image  and  return  the  coordinates  of  the  centroid 
a  shape,  along  with  an  orientation  measurement  that  can  be  used  to  create  a 
ansformation  to  locate  an  actual  piece.  The  coordinates  are  returned  in 
its  of  pixels  and  it  is  the  users  responsibility  to  scale  these  pixels 
to  actual  coordinates. 


e  orientation  value  is  returned  in  radians  and  it  must  be  converted  to  degrees 
fore  being  used. 


riables  Description: 


golor  -  this  variable  has  two  legal  values,  negative  one  and  zero.  If  the  user 
sets  this  variable  to  negative  one  white  shapes  will  be  used.  If  the 
the  user  sets  it  to  zero  the  vision  system  will  use  black  shapes. 


expected  .shapes  -  this  variable  will  tell  the  vision  system  how  many  matches 
to  allow. 


fast  -  this  variable  will  tell  the  vision  system  whether  or  not  the  action  is 
to  occur  in  a  fast  mode  of  operation  or  not.  If  the  user  selects  fast 
by  setting  this  variable  to  one  the  user  must  recall  this  routine 
with  this  switch  set  to  zero  to  collect  the  data.  .  ,  ,  . 

/U^> 


tew. picture  -  this  variable  instructs  the  vision  system  to  -1  repicture, 
0  for  previous  picture  and  1  for  picture. 

j..t —  •  -  o  i* //■»■* 

lumblob.id  -  this  variable  contains  the  number  of  prototypes  to  find,  if 
you  want  the  vision  system  to  locate  all  of  prototypes  set 
this  variable  to  -1. 


jlob.theref]  -  this  variable  indicates  in  a  loqical  fashion  if  the  prototype 
is  in  the  field  of  view,  if  the  prototype  is  in  the  current 
picture  this  variable  will  be  set  egual  to  TRUE.  If  the 

not  in  the  field  of  view  the  variable  will  be 


prototype  is 

set  to  FALSE.  The  element  number  is  eauated  to  the  prototype 
number. 

this  variable  indicates,  in  a  logical  fashion,  the  result  of  the 
execution  of  the  routine.  If  it  returns  TRUE,  it  indicates  that 
an  error  was  detected  by  the  vision  system.  The  actual  error  code 
will  be  found  in  the  variable  s. system. error.  If  it  returns  a 
FALSE,  the  routine  completed  without  an  error. 


s .messaged  -  reserved  for  system  use 
s.numbytes  -  reserved  for  system  use 
s.numwords  -  reserved  for  system  use 

s. orient. blob  (J  -  Contains  the  angular  measurement  of  the  prototype 

whose  number  is  the  array  offset. 

s. system. error  -  This  variable  contains  the  actual  error  code  returned 
from  the  vision  system.  It  should  be  referenced  when 
the  variable  s. error  indicates  that  a  problem  occured 
during  the  execution  of  the  routine. 
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If  s. error  ia  FALSE,  this  variable  will  be  equal  to  xero. 

s.xcenter.blob  [)  -  Contains  the  value  in  pixels  locating  the  prototype 

whose  number  is  the  array  offset,  in  the  X  direction. 

s .ycenter .blob  (1  ~  Contains  the  value  in  pixels  locating  the  prototype 

whose  number  is  the  array  offset,  in  the  Y  direction. 


Example:  For  prototype  one  in  the  picture,  its  X  location  will  be  found 

in  variable  S.xcenter .blobfl ]  ,  the  Y  location  will  be  found  in 
variable  S. ycenter. blobfl ] ,  and  finally  the  orientation  will  be 
found  in  S .orient .blob [ 1 1 .  The  number  found  between  the  brackets 
is  the  prototype  number.  If  an  unidentified  shape  is  in  the 
field  of  view,  then  it  will  be  referenced  by  array  offset  zero. 
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POSSIBLE  VALUES  FOR  S . SYSTEM . ERROR 
Value  Meaning 

Corrective  Action  for  the  following  errors: 

Examine  the  variables  that  you  are  reguired  to  set. 
Reset  the  bad  one  to  a  legal  value. 


1  Bad  Parameter  has  been  passed  to  the  routine.  The  Routine 

has  not  executed . 

Corrective  Action  for  the  following  errors: 

None,  a  zero  indicates  a  good  run. 


0  The  routine  ended  with  out  any  error  detected  by  the  vision 

system. 

Corrective  Action  for  the  following  errors: 

Reattempt  the  communication  with  the  vision  system. 

Attempt  to  reload  both  the  vision  system  and  the  VAL-II  system, 
if  this  error  persists  please  note  all  actions  that  lead  up 
to  this  and  contact  Unimation  Inc. 


-1 


-3 


-5 


-7 


-17 


-21 


-23 

-129 

-131 

-133 

-135 


Not  yet  implemented.  The  command  number  passed  to  the  vision 
system  is  not  defined. 

The  command  number  is  out  of  range. 

The  argument  passed  to  the  vision  system  is  out  of  range. 

The  command  number  is  reserved  for  future  use. 

The  variable  name  passed  to  the  vision  system  was  not  recieved 
properly. 

Bad  Restart  operation,  this  error  should  only  occur  during 
the  execution  of  the  boot  routine. 

Vision  System  restarted  during  command. 

Error  from  User  routine. 

User  routine  number  out  of  range 

Buffer  to  small  for  return. 

No  User. 


Corrective  Action  for  the  following  errors: 
Rename  the  one  of  the  prototypes. 
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-19 


Duplicate  prototype  name. 


Corrective  Action  for  the  following  errors: 

This  collection  of  errors  most  probably  result  from  specifying 
an  illegal  blob  number. 

-9  Blob  is  not  active. 

-11  Blob  Specification  is  illegal. 

-13  No  Blob  found  . 


Program:  U. Saver 
Instructions/Usage 

U. Saver  is  a  menu  driven  program  that  allows  you  to  delete  prototypes  from  the 
vision  system,  copy  them  from  the  vision  system  to  val ,  and  copy  them 
from  val  to  the  vision  system.  It  deals  with  only  prototype  information  not 
orientation  features,  or  other  switches. 

To  upload  prototypes  select  option  1,  then  enter  the  prototype  number. 

Upon  completion,  store  the  real  variable  PROTOINFOM  to  disk. 

To  download  first  load  the  real  variables  from  disk  and  then  execute  the 
program.  Select  option  2  and  the  prototype  will  be  restored. 

To  delete  a  prototype  select  option  3,  then  enter  a  1  to  confirm. 


Program:  U.  train. blob 
Instruct iona/Usage 

U. Train. blob  will  allow  you  to  add  in  a  blobs  description  to  a  prototypes 
description,  (this  is  called  training)  It  will  use  the  primary  blob. 

The  user  simply  enters  the  prototype  number  and  then  the  number  of  training 
times.  By  depressing  <REC>  on  the  pendant  a  picture  will  be  taken,  and  the  blob 
will  be  added  into  the  prototype. 


Note:  This  program  calls  the  user  routine  u .take .picture  and  sets  the  flag 
u .picture . fast  to  zero  (slow  mode).  This  routine  will  not  function 
reliably  unless  the  vision  system  has  been  initialized. 
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Appendix  B 


VRS  Software  Program  Listings 

This  Appendix  contains  the  aoftware  program  listings  for 
the  vision-robot  system  (VRS).  The  listings  are  grouped 
together  by  the  tasks  in  which  they  were  accomplished. 

A  miniature  Table  of  Contents  of  the  program  listings 
follow : 


Table  of  Contents 


Task  Page 

1.  Static  Vision  Servo  Control  Program  Listings  ....  125 

vision. menu  .  126 

vis  ion . robot . cal  .  128 

prototype . train  .  131 

blob . store . or . de lete  .  132 

vi s ion . demo . menu  .  133 

ident. blobs  .  134 

locate  .  135 

ident . blob . point . center  .  136 

camera . center . of f set  .  137 

point. center  .  138 

track. targ  .  139 

center  .  140 

t rack . targ . scat  .  141 

center. scat  .  143 

search  . .  145 

search. track  . 147 

search . center  . 149 

2.  Dynamic  Vision  Servo  Control  Program  Listings  ....  152 

track . targ . dcat  . 153 

center. dcat  .  155 
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Task  1:  Static  Visual  Servo  Control  Program  Listings 


125 


i 


rl  vision. menu 
PROGRAM  vision. menu 

1  ; 

2  ;  THIS  PROGRAM  PROVIDES  A  MENU  FOR  SELECTING  VARIOUS  OPTIONS 

3  ",  WHEN  USING  THE  VISION  SYSTEM.  MOST  OF  PROGRAMS  AVAILABLE  , 

A  ",  ARE  SYSTEM  PROGRAMS  FROM  THE  UNI VISION1 . PG  FLOPPY  DISK. 

5  ;  THERE  ISN'T  MUCH  DOCUMENTATION  AVAILABLE  FOR  THESE  PROGRAMS  r 
G  ;  HOWEVER  THEY  ARE  FAIRLY  SIMPLE  AND  USER  FRIENDLY. 

7  ; 

8  ",  THE  VISION/ROBOT  SYSTEM  MUST  BE  INITIALIZED  BEFORE  EXECUTING 

9  ;  ANY  PROGRAMS. 

10  ;  THE  VISION/ROBOT  INTERFACE  IS  INITIALIZED  BY  CALLING 

11  ;  U. INITVISION. 

12  : 

1 3  '  ** **•»*■»•»*■»■**■»#•* ***•**■#•* WARN  I  NCi****** *****************  ***■#•* 

14  5  *  * 

15  J  *  ONCE  THE  INTERFACE  HAS  BEEN  INITIALIZED  FOR  THE  * 

1G  ;  *  1ST  TIME,  THE  VISION/ROBOT  SYSTEM  MUST  BE  CALIBRATED.  * 

17  ;  *  * 

IB  ",  in*#****#***#*#*#########*###*###*###***#**##*##*#*##*#*## 


CAUTION 


IF  THE  CAMERA  IS  EVER  MOVED  RELATIVE  TO  THE  ROBOT,  THE 
REFERENCE  FRAME  FOR  OBJECTS  VIEWED  BY  THE  VISION  SYSTEM 
WILL  ALSO  BE  MOVED.  IN  ORDER  FOR  THE  ROBOT  TO  MOVE  TO 
LOCAT I ONS  DEFINED  BY  THE  VISION  SYSTEM,  A  RELATIVE 
TRANS FROM AT ION  MUST  BE  DEFINED  WHICH  RELATES  THE  REFERENCE 
FRAME  OF  THE  VISION  SYSTEM  TO  THAT  OF  THE  ROBOT ", 
vision. robot. cal  WILL  ACCOMPLISH  THIS  TASK. 


TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

num 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 


/  B ,  /CZ'r  beeps  the  screen  &  skips  2  lines 

"*fr#**#**»*4*****»***#*************#********#*#" 
"  *  #  " 

"*  WELCOME  TO  AFIT'S  VISION  LAB  *" 

"  *  *  " 

"  1HHHt«***4**<HHHHt44*<HHHHt#*****#*******4*-»**#*4  " 

:  01  initialize  parameter 
/C2,  /B 

"  THE  FOLLOWING  OPTIONS  ARE  AVAILABLE 
/Cl 

"  1.  INITIALIZE  VISION/ROBOT  COMMUNICATION  ", 

"  2.  V I S I  ON- TO- ROBOT  CALIBRATION  ",  /Cl 

"  3.  PROTOTYPE  TRAINING  ",  /Cl 

"  4.  PROTOTYPE  STORAGE  AND  DELETION  ",  /Cl 

"  5.  VISION  DEMO  ",  /Cl 


TYPE  "*««««»«**»*«««*«*««»**4«»*»«**4**««****»«««*" 

TYPE  "#  *" 

TYPE  "*  - >  WARNING  < -  *" 

TYPE  "*  *" 

TYPE  "*  BEFORE  EXECUTING  OPTIONS  3,4,  OR  5,  *" 

TYPE  "*  OPTIONS  1  AND  2  MUST  BE  ACCOMPLISHED  *" 

TYPE  "*  *" 

TYPE  "**•****■»*************#*■»*#*******•»■»■*■»**•»*****",  /Cl 

TYPE  "PLEASE  ENTER  THE  NUMBER  OF  THE  OPTION  " 


PROMPT 


YOU  DESIRE  - > 


,  num 
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62 

G3 

G4 

65 

6G 

67 

60 

G9 

70 

71 

72 

73 

74 

75 

76 

77 
70 
70 
GO 


CASE  INT(num)  OF 

value  i: 

CALL  u.initvisionJ  INITIALIZE  VISION/ROBQT  SYSTEM 
TYPE  /C2 

PROMPT  •*  =  =  =  >  I  NT  I AL I ZAT I  ON  COMPLETE  »  PRESS  <RETURN>  TO  CONTINl 
VALUE  2: 

CALL  vision. robot. cal  '•  PROGRAM  TO  CALIBRATE  SYSTEM 
VALUE  31 

CALL  protot  v-pe .  train USE  THIS  TO  TEACH  THE  VISION  SYSTEM 
;  DIFFERENT  OBJECTS 

value  4: 

CALL  blob .store . or . delete  I  THIS  WILL  STORE  OR  DELETE  A 
;  PROTOTYPE  INTO/FROM  THE  VISION  SYSTEM'S 
;  MEMORY. 


81  VALUE  51 

82  CALL  v i s i on . demo . menu J  THIS  EXECUTES  THE  VISION  DEMO 

03  ANY 

84  TYPE  "THAT  NUMBER  IS  NOT  AN  OPTION  PLEASE  TRY  AGAIN" 

05  GOTO  10 

8G  END 

87  ; 

08  ; 

89  GOTO  10 

90  RETURN 


Pi  vision. robot. cal 
PROGRAM  vision. robot. cal 
1  ; 

5  •  lSiS^T20GRAM  E>''ECUTES  THREE  system  programs. 

“  :  ™f/X,?8T  PROCRAM*  u .  boot  f  REBOOTS  THE  VISION  SYSTEM. 

I  ,  THIS  HAS  THE  EFFECT  OF  DELETING  ALL  THE  PROTOTYPES  FROM 

5  ,  MEMORY.  THEREFORE  NEW  PROTOTYPES  MUST  BE  TRAINED. 

6  * 

7  J  THE  SECOND  PROGRAM »  u.K. teach,  DETERMINES  THE 
S  ;  SCALE  FACTOR  (s.K. scale)  USED  IN  CALIBRATING 
9  m,  THE  V ISION-TO-ROBQT  SYSTEM.  THIS  PROGRAM 

10  RELATES  THE  CAMERA'S  CHARACTERISTICS  <ie.  FOCAL 

11  '  LENGTH)  TO  CAMERA  PIXEL  SIZE.  IT  DEFINES  THE  RATIO  OF 

12  I  THE  DISTANCE  BETWEEN  2  LOCATIONS  IN  ROBOT  COORDINATES 

13  !  TO  THE  DISTANCE  IN  PIXEL  COORDINATES. 

14  ; 

15  ;  A  GOOD  RANGE  FOR  s.K. scale  IS  APPROXIMATELY 

IS?  1 <  s . k . s  c  a  1 e  <  2 

17  J  where  the  units  are  in  mm. 

18  ; 

19  J  THE  THIRD  SYSTEM  PROGRAM  CALLED  IS  u . frame . t e ac h 

20  ;  WHICH  ESTABLISHES  THE  ROBOT/VISION  WORK  FRAME. 


CALL  u.boot;  REB001  -*•**.  I  SIGN  SYSTEM 


TYPE  /B,  /CB 

TYPE  " I  I 

TYPE  "I  THE  FOLLOWING  PROGRAMS  WILL  ESTABLISH  THE  I 

TYPE  " !  ROBOT-VISION  WORK  AREA  I 

TYPE  " I  I 

TYPE  /C2 


TYPE  "  STEP  1  — >  SELECT  THE  o  VIEW  CURRENT  CAMERA,  OPTION",  /Cl 
TYPE  "  STEP  2  — >  FOCUS  THE  CAMERA  USING  THE  CALIBRATION- 
TYPE  "  DISK  AS  A  GUIDE",  /Cl 

PROMPT  "======  PRESS  <  RETURN.-*  TO  CONTINUE  ======" 

CALL  u.K. teach!  SYSTEM  PROGRAM  USED  TO  DETERMINE  s.K. scale 

TYPE  /Cl,  /B 

TYPE  "A  GOOD  VALUE  FOR  S.K. SCALE  IS  APPROXIMATELY",  /Cl 
TYPE  "  1  <  s.K. scale  <  2  ",  /Cl 


PROMPT  "======  PRESS  <RETURN>  TO  CONTINUE  == 


CALL  u. Frame . teach J  SYSTEM  PROGRAM  TO  WORK  FRAME 


TYPE  /B,  /Cl 

TYPE  "========================== ===r======r====== 

TYPE  " I  I 

TYPE  "I  NOW  A  CHECK  OF  THE  VISION-TO-ROBDT  I 

TYPE  "I  TRANSFROMATION  WILL  BE  VERIFIED  AND  I 

TYPE  "I  MODIFIED,  IF  NECESSARY.  I 

TYPE  " I  I 

TYPE  "  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =:  =  =  =  =  =  =  =  =  =  = 
TYPE  /B,  /Cl 
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E2  ; 
S3  ; 
Q4  ; 

65  ; 

66  ; 

67  ; 

68 

68  ; 

70 

71 

72  ; 

73 

74 

75  ; 

76  ; 

77  ; 

78  ; 

79  ; 

80 
81 
82 

83 

84 

85 

86  ; 

87 

88 
88 

90 

91 

92 

93 

94 

95 
9G 
97 
93 
99  ; 


MOVE  #s. safe. position*  MAKES  SURE  ARM  IS  OUT  OF  CAMERA'S 


TYPE  "*= 
TYPE  "  =  = 


PLACE  THE  CALIBRATION  DISK  IN  THE  APPOXIMATE 
CENTER  OF  THE  CAMERA'S  FOV 


PROMPT  "*■■■■ *  PRESS  <RETURN>  TO  CONTINUE  = 
TYPE  /Cl 


THE  FOLLOWING  WILL  INITIALIZE  PARAMETERS  REQUIRED  IN 
THE  REST  OF  THIS  PROGRAM. 

K.Off  =  0 

v .  o  F  f  =  0 
2. off  =  0 
xc  =  0 
vc  =  0 

2  C  =  0 

CALL  u. take. Picture!  TAKE  A  PICTURE 
CALL  u.recoani2e;  IDENTIFY  OBJECTS 
x  =  s. proto. id 

CALL  u . wh er e . are . you . pro t o ;  ANALIZES  IMAGE  AND  PROVIDES 

;  ITS  x  &  y  CENTROID  POSITIONS 
xcent  =  s.k.scale*s.xcenter.blobCx3 
vcer.t  =  s.K.scale*s.  v center. b lob  Cx3 

SET  center  =  camera  TRANS  (  x  c  en  t  »  /cent.  0.  0.  -90.  0) 

;  camera  IS  THE  THE  RELATIVE  TRANSFROMATION 
J  DETERMINED  WHEN  u . frame . t eac h  WAS  EXECUTED. 

SPEED  100 
MOVE  center 


100 

101 

102 

103 

104 

105 

106 

107 

108  ; 

109 

110 
111 
112 

113 

114 

115 

116 

117 

118  ; 

119 

120 
121 
122 

123  ; 

124  ; 

125  ; 

126  ; 

i?7  : 


TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 


"I  LOOK  AT  THE  POSITION  OF  THE  ROBOT 
"I  POINTER  TO  THAT  OF  THE  CALIBRATION 
"I  DISK.  IF  IT  IS  NOT  IN  THE  CENTER 
"I  ANSWER  THE  FOLLOWING  QUESTIONS 


/B.  /Cl 


TYPE  "THE  POINTER  IS  OFF  CENTER  BY  +  or  -  _  mm's" 

PROMPT  "  IN  THE  'X'  DIRECTION  x.off 

TYPE  /Cl 

TYPE  "THE  POINTER  IS  OFF  CENTER  BY  +  or  -  _  mm's" 

PROMPT  "  IN  THE  'Y'  DIRECTION  — >",  y.off 
TYPE  /Cl 


TYPE  "THE  POINTER  IS  TOO  FAR  ABOVE/BELOW  THE  TABLE  BY  " 

PROMPT  "  +  or  - _ mm's  — >"»  z.off 

TYPE  /Cl 


CALL  camera. center. oFFset?  THIS  WILL  CORRECT  THE  OFFSET 
SET  spot  =  cameral l TRANS ( xcent .  ycent.  0.  0.  -90.  0) 
SPEED  25 
MOVE  spot 


FOV 

.  /Cl 
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12B  ; 

129  : 

130 

131 

132 

133 

134 

135 
13G 

137 

138 

139 

140 

141 

142  ; 

143 

144 

145 
14G 

147  ; 

148 

149 

150 

151  ; 

152 

153  ; 

154 

155  ; 
15G 

.END 


TYPE  /B 

TYPE  "»»>  CHECK  THE  POINTER  W.R.T.  THE  DISK  AGAIN  <«*" 
TYPE  /Cl 

TYPE  "THE  POINTER  IS  OFF  CENTER  BY  +  or  -  _  mm's" 

PROMPT  "  IN  THE  'X'  DIRECTION  — >" ,  xc 
TYPE  /Cl 

TYPE  "THE  POINTER  IS  OFF  CENTER  BY  +  or  -  _  mm's" 

PROMPT  "  IN  THE  'Y'  DIRECTION  — >"r  y c 
TYPE  /Cl 

TYPE  "THE  POINTER  IS  TOO  FAR  ABOVE/BELOW  THE  TABLE  BY" 

PROMPT  "  +  or  -  _  mm's  zc 

TYPE  /Cl 

CALL  camera . cent er . oFFs et 

SET  spot  =  camera  1  I  TRANS ( x c en t f  ycenti  0»  0>  -90»  0) 
SPEED  25 
MQUE  spot 

TYPE  "THIS  SHOULD  BE  CLOSE  ENOUGH »  THE  SYSTEM  IS  NOW" 
TYPE  "  READY  FOR  YOUR  CON'.’IENCE!  !!!!!!!!" 

TYPE  /Cl 

PROMPT  "======>  PRESS  < RETURN >  TO  CONTINUE  <======" 

READY 

CALL  vision. menu!  GO  BACK  TO  MAIN  MENU 


i 

( 
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i 


pi  prototype. train 


PROGRAM  protot ype . train 

i  ; 


•L  W 

3  ; 

4  ; 

5  ; 

g  ; 

7  ; 

e  : 
s  ; 

10  ; 

11  : 

12 

13 

14 

15 
1G 

17 

18 

19  ; 

20 

21 
«-» r? 

.END 


THIS  PROGRAM  CALLS  u. train. blob  TO  TEACH  THE  VISION 
SYSTEM  THE  VARIOUS  PROTOTYPES  THE  USER  IS  INTERESTED 
IN  RECOGNIZING. 

FOR  THE  VISION  DEMO  OPTION,  PROTOTYPES  1  AND  2  SHOULD  BE 
TRAINED  AS  A  CIRCLE  AND  A  RECTANGLE,  RESPECTIVELY.  THE  REASON 
FOR  THIS  CONSTRAINT  LIES  IN  THE  FACT  THAT  THE  VISION 
DEMO  HAS  PROGRAMS  WHICH  ARE  BASED  ON  THE  PROTOTYPES  BEING 
DEFINED  WITH?  1  —  CIRCLE,  2  —  RECTANGLE. 

TYPE  /Cll 

TYPE  "  ===>  NOTE  <===  " 

TYPE  /e,  /Cl 

TYPE  "  IF  THE  USER  IS  TRAINING  OBJECTS  FOR  THE  VISION  DEMO- 
TYPE  /Cl 

TYPE  "  =  =  >  PROTOTYPE  1  SHOULD  BE  A  CIRCLE  <  =  ,  /Cl 
TYPE  "==>  PROTOTYPE  2  SHOULD  BE  A  RECTANGLE  <==",  /C3 

PROMPT  *•  =  =  =  =  =  =  >  PRESS  < RETURN >  TO  CONTINUE  <=  =  =  =  =  =  " 

CALL  u. train. blob 

CALL  vision. menu;  GO  BACK  TO  MAIN  MENU 
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•  p1  b 1 ob . s tore . or. de  1  ete 

•  PROGRAM  b 1 ob . s t ore . or . de  1  e t e 

1  ; 


2  ;  THIS  PROGRAM  IS  USED  WHEN  STORING  OR  DELETING  PROTOTYPE 

3  J  INFORMATION  INTO/FROM  THE  VISION  SYSTEM'S  MEMORY. 

4  ; 


5  '  u . saver  IS  A  MENU  DRIVEN  SYSTEM  PROGRAM  WHICH  IS  CALLED 
G  ;  WHEN  THE  USER  WISHES  TO  COMPLETELY  DELETE  A  PROTOTYPE  FROM 

7  I  THE  VISION  SYSTEM,  OR  TO  COPY  THEM  FROM  THE  VISION  SYSTEM 

8  I  TO  VAL  II  AND  VIS  VERSA.  u. saver  DEALS  ONLY  WITH  PROTOTYPE 

9  !  INFORMATION,  NOT  ORIENTATION  FEATURES,  OR  OTHER  SWITCHES. 

10  ; 

11  ;  u. delete. blob  WOULD  BE  USED  IF  THE  USER  WISHED  TO 

12  !  DELETE  ONLY  A  PROTOTYPE'S  FEATURES  FROM  MEMORY. 


13  ; 

14  ; 

is  ; 
is  ; 

17  ; 

18 

19 

20 
21 

23 

24 

25 

26  ; 

27 

28 

29 

30 

31 

32 

33 

34 

35 
3S 
37 
3S 

39 

40 

41 

42 

43 

44  ; 

45 
.END 


u.boot  REBOOTS  THE  VISION  SYSTEM.  THIS  HAS  THE  EFFECT  OF 
DELETING  ALL  THE  PROTOTYPES  FROM  MEMORY. 


15  TYPE  /e,  /CU 

TYPE  "THE  FOLLOWING  OPTIONS  ARE  AVAILABLE",  /Cl 
TYPE  "  1.  STORE,  RECALL,  OR  DELETE  A  PROTOTYPE  ",  /Cl 

TYPE  "  2.  DELETE  ONLY  A  PROTOTYPE'S  FEATURES  ",  /Cl 

TYPE  "  3.  DELETE  ALL  PROTOTYPES  FROM  MEMORY  ",  /Cl 
TYPE  "  4.  GO  BACK  TO  THE  VISION  LAB  MAIN  MENU",  /Cl 
TYPE  "PLEASE  ENTER  THE  NUMBER  OF  THE  OPTION  " 


PROMPT  "  YOU  DESIRE  — > 


,  n  urn 


CASE  INT(num)  OF 
VALUE  i: 

CALL  ul saver?  STORE,  COPY,  OR  DELETE  A  PROTOTYPE 
VALUE  2: 

TYPE  /e 

TYPE  "ENTER  THE  NUMBER  OF  THE  PROTOTYPE  WHOSE  FEATURES 
PROMPT  "  YOU  WISH  TO  DELETE  — >  ",  u. delete. blob. num 
TYPE  /Cl 

CALL  a. delete. blob ;  DELETE  A  BLOB'S  FEATURES 
VALUE  3*. 

CALL  u.boot;  DELETE  ALL  PROTOTYPES  FROM  MEMORY 
VALUE  4: 

CALL  vision. menu?  GO  BACK  TO  MAIN  MENU 

ANY 

TYPE  "THAT  NUMBER  IS  NOT  AN  OPTION,  PLEASE  TRY  AGAIN" 
GOTO  15 

END 

CALL  vision. menu?  GO  BACK  TO  MAIN  MENU 


II 
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M, 


•p1  v i s x on . demo . menu 
•PROGRAM  u i 5 i on . demo . menu 
1  ! 


3  ; 

4  ; 

5  ; 

6  ; 
7  ; 

a 

9 

10 
1 1 
12 

13 

14 

15 

16  ; 
17 

is  ; 

19 

20 
21 
22 

23 

24  ; 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34  ; 

35 

36 

37  ! 

38 

39 

40 

41 

42 

43 

44  ' 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54  ; 

55  ; 
5G 
57 

•  END 


Ip  utct^RS^oR°VIDES  A  MENLJ  F0R  SELECTING  VARIOUS  CHOICES 

EACH  TIME  THE  HENU  PROSRAM  IS  EXECUTED?3 
SVSTPH  INTERFACE  IS  INITIALIZED  BY  CALLING  A 

a i b T tM  PROGRAM,  u.xnxtvision.  (LOCATED  ON  THE  UNIVISI0N1  PG 
FLOPPY  DISK).  '  u 


CALL  u.  xnituision!  INITIALIZE  THE  VISION/ROBOT  INTERFACE 
u. new. Picture  *  i;  SYSTEM  LOOKS  AT  CURRENT  PICTURE 
u. numb  lob. id  =  -II  VISION  SYSTEM  WILL  IDENTIFY  ALL  OBJECTS  IN  FOV 
u.n urn blob  =  1 !  BLOB  1  WILL  BE  COMPARED  TO  ALL 
!  KNOWN  PROTOTYPES 

u.bloonum  *  2!  set  this  number  to  the  max  number  of 

-  of  blobs  expected  in  the  camera's  FOV. 
u . p i c t ur e . f as t  =  0?  set  slow  mode 

10  TYPE  /E,  /C3!  BEEPS  THE  SCREEN  AND  SKIPS  3  LINES 

TYPE  " **************************  *****  ******  ♦■fr*******1' 

TYPE  "*  *•• 

TYPE  "*  WELCOME  TO  AFIT'S  VISION  DEMO  *» 

TYPE  "*  * " 

TYPE  "**********************************************" 


SPEED  755  robot  speed  is  set  to  757.  of  its  normal  value 
READY 


TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 


/Cl.  /B 

"  THE  FOLLOWING  DEMOS  ARE  AVAILABLE 
/Cl 

“  1.  OBJECT  IDENTIFICATION  ",  /Cl 

"  2.  OBJECT  IDENTIFICATION  AND  ROBOT  POSITIONING 

"  3.  DIRECT  ROBOT  POSITIONING  ",  /Cl 

"  4.  GO  BACK  TO  THE  VISION  LAB  MAIN  MENU",  /Cl 


/Cl 


TYPE  "PLEASE  ENTER  THE  NUMBER  OF  THE  DEMO  " 
PROMPT  "  YOU  WISH  TO  VIEW  - >  ",  num 


CASE  INT(num)  OF 
VALUE  i: 

CALL  ident. blobs!  PROGRAM  TO  IDENTIFY  BLOBS 
VALUE  21 

CALL  ident. blob. point. center!  PROGRAM  TO  IDENTIFY  BLOBS, 

!  &  DIRECTS  THE  ROBOT  TO  MOVE  TO  THE  CENTER  OF  THE 
!  LARGEST  BLOB  IN  THE  CAMERA'S  FIELD  OF  VIEW  (FOV) 
VALUE  3! 

CALL  point. center!  DIRECTS  THE  ROBOT  TO  MOVE  TO  THE  CENTER 
!  TO  THE  LARGEST  BLOB  IN  THE  CAMERA'S  FOV. 

VALUE  4! 

CALL  u i s i on .menu !  GO  TO  MAIN  MENU 
ANY 

TYPE  "THAT  NUMBER  IS  NOT  AN  OPTION  PLEASE  TRY  AGAIN" 

GOTO  10 

END 


PROMPT  "=«====  PRESS  <RETURN>  TO  CONTINUE  =****=" 
GOTO  10 
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•p1  ident. blobs 
.PROGRAM  idem,  blobs 
i  ; 


2  i 

3  ; 

4  ; 

5  ' 

g  ; 

7  ; 

e  ; 

9  ; 

10  ; 

11 

12 

13 

14 

15 
1G 

17 

18 

19 

20 
21 
22 

23 

24 

25 


THIS  PROGRAM  CALL  A  SYSTEM  PROGRAM,  u. take . Picture . 

WHICH  ASKS  THE  VISION  SYSTEM  TO  TAKE  A  PICTURE. 

THIS  PROGRAM  PROVIDES  THE  NUMBER  OF  BLOBS  IN  THE 
CAMERA'S  FOV. 

AFTER  COMPLETION  OF  THIS  PROGRAM,  u. recognize,  IS 
CALLED  TO  IDENTIFY  THE  OBJECTS  OBSERVED  WHEN  THE  LAST 
PICTURE  WAS  TAKEN. 

CALL  u. take. Picture;  ASK  THE  VISION  SYSTEM  TO  TAKE  A  PICTURE 
nb  =  5. blob. count 
TYPE  /B,  /Cl 
IF  s. error  THEN 

TYPE  "BAD  ROUTINE  -  ERROR  CODE  IS  — >" ,  /D,  s . 5 ys tew . error 

ELSE 

TYPE  "GOOD  PICTURE  -  THERE  ARE  ",  /D,  nb 
TYPE  /B.  " BLOB ( S )  IN  THE  CAMERA'S  VIEW!" 

END 

CALL  u. blink. blob;  OUTLINE  THE  BLOB  IN  WHITE 
CALL  u. recoanize;  IDENTIFY  THE  OBJECTS 
TYPE  /B 
TYPE  /Cl,  " 

x  =  s.proto.idl  SET  NUMBER  OF  THE  LARGEST  PROTOTYPE 
;  IDENTIFIED. 


26  ; 
27  ; 

za  ; 

29  ; 

so  ; 

31  ; 

32  ; 

33  ; 

34 

35 
3G 

37 

38 
38 

40 

41 

42 

43 

44 

45 
4G 

.END 


IT  SHOULD  BE  NOTED  HERE  THAT  THE  USER  MUST  PREVIOUSLY  TRAIN 
THESE  OBJECTS,  WITH  THE  APPROPRIATE  NUMBER,  TO  THE  VISION 
SYSTEM.  FOR  EXAMPLE,  OBJECT  "1"  IS  TRAINED  TO  BE  A  CIRCLE 
AND  OBJECT  2  IS  TRAINED  TO  BE  A  RECTANGLE. 

ALSO  IF  AN  OBJECT  IS  NOT  RECOGNIZED  BY  THE  SYSTEM 
-  s . proto .id  =0  - 


CASE  INT(x)  OF 


VALUE  o: 
TYPE  "* 

I  DON'T  RECOGNIZE  i 

ANY 

OBJECTS 

* 

value  i: 
TYPE  “* 

THE 

LARGEST 

OBJECT 

IS 

A 

CIRCLE 

* 

VALUE  2 1 
TYPE  "* 

THE 

LARGEST 

OBJECT 

IS 

A 

RECTANGLE 

* 

END 

TYPE  "*«************** ******* *•***##•**  ******" 
TYPE  /Cl 

CALL  locate!  ASK  THE  VISION  SYSTEM  TO  GIVE  THE 
;  CENTROID  POSITION  OF  THE  OBJECT. 

RETURN 
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.pi  locate 
•PROGRAM  locate 
1  ; 

r  2  ;  THIS  PROGRAM  CALLS  u .where . are . you. proto  TO  DETERMINE 
w  3  J  AN  OBJECTS  CENTROID  AND  ORIENTATION. 

4  ; 

5  J  THE  x  4  y  COORDINATES  ARE  SCALED  BY  THE  SCALE  FACTOR,  s.k.  scale, 

G  J  WHICH  DEFINES  THE  RATIO  OF  THE  DISTANCE  BETWEEN  2  LOCATIONS 

7  J  IN  ROBOT  COORDINATES  TO  THE  DISTANCE  IN  PIXEL  COORDINATES. 

8  ; 

9  CALL  u . where . are . you . pro ta 

10  xcent  *  c  .  K . s ca 1 e*s . xc en t er . b 1 o b C x 3 

11  ycent  =  s  .  K . sea  1 e#s . ycen t er . b 1 o b C x 3 

12  orien  =  180*s . or i ent . b 1 ob C x 3 /PI 

13  TYPE  "THE  X  COORDINATE  OF  THE  CENTROID  (in  mms)  IS",  /D,  xcent 

14  TYPE  /Cl 

15  TYPE  "THE  Y  COORDINATE  OF  THE  CENNTROID  (in  mms)  IS",  /D.  ycen 

1G  TYPE  /Cl 

17  TYPE  "THE  ORIENTATION  IS",  /D,  orien 

18  TYPE  /C2 

19  RETURN 
•  END 
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.p1  i den t . b 1 ob . po inx . c enter 
•PROGRAM  i den t . b 1 ob . po i n t . c ent er 
i  ; 


2  ; 

3  ; 

4  ; 

5  ; 

6  ; 

7 

8 
9 

10 
1 1 
12 

13 

14 

15 
IB 

17  ; 
is  ; 
is  ; 
20 

21 
n  n 

±iL 

23 
•  END 


THIS  PROGRAMS  CALLS  VARIOUS  PROGRAMS  TO  ENABLE  THE 
VISION  SYSTEM  TO  IDENTIFY  AN  OBJECT  AND  PROVIDE 
THE  APPROPRIATE  COMMANDS  AND  COORDINATES  FOR  THE  ROBOT 
TO  POINT  TO  THE  CENTROID  OF  THE  OBJECT. 

MOVE  #s. safe .position?  MOVE  THE  ARM  TO  A  POSITION 
;  OUTSIDE  THE  CAMERA 'A  FOV. 

TYPE  /B,  /C2 

CALL  ident. blobs?  VISION  SYSTEM  TAKES  A  PICTURE 
;  IDENTIFIES  BLOBS  AND  PROVIDES  THE 
;  OBJECTS  X  AND  Y  CENTROID  POSITION. 

CALL  camera . o en ter . oFF s e t ;  THIS  PROGRAM  PROVIDES  THE 
J  CORRECTION  FOR  THE  OFFSET 
;  BETWEEN  THE  OBJECT'S  CENTROID 
;  AND  THE  POSITION  OF  THE  POINTER 

THE  NEXT  EXPRESSION  DEFINES  THE  LOCATION  OF  THE  OBJECT 
IN  THE  ROBOT  REFERENCE  FRAME. 

SET  spot  =  camera  1 ITRANS(xcent»  ycent,  0»  orien,  -90. 

SPEED  75 

MOVE  spot!  MOVE  ARM  TO  THE  DESIRED  LOCATION 

RETURN 


0) 


f 


136 


c  i  n  in  id  P' 


.  p1  camera. center. offset 
.PROGRAM  cam era. center. offset 
1  ; 

!  THIS  PROGRAM  SIMPLY  PROVIDES  THE  CORRECTION!  FOR  THE  OFFSET 
J  BETWEEN  THE  OBJECT'S  CENTROID  AND  THE  POSITION  OF  THE  ROBOT 
'  POINTER. 

r 

SET  camera!  -  SHIFT(camera  BY  x.off+xc.  v.off+ycr  z.off+zc) 
RETURN 

.END 
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•Pi  point. center 
.PROGRAM  point. center 
i  ; 


Z  ;  THIS  PROGRAM  IS  BUILT  FOR  SPEED.  IT  SIMPLY 
2  ;  INSTRUCTS  ThE  VISION  SYSTEM  TO  TAKE  A  PICTURE,  BY 

4  J  CALLING  u. take. Picture.  THE  PROGRAM  THEN  IDENTIFIES 

5  ;  THE  OBJECTS  USING  u. recognize.  AND  FINALLY,  THE 

G  J  OBJECT'S  CENTROID  IS  DETERMINED  BY  u.where.are. /ou.proto. 


7 

8 

10 
i  i 

1  z 

13 
1  4 
15 
1G 
1  7 
IS 
i  s 

Z 1 

>~y 

Z-a 


10  MOVE  #s. safe. position!  MOOES  ARM  TO  A  POSITION 
OUTSIDE  THE  CAMERA'S  FOO. 

CA_L  u. take. Picture;  VISION  SYSTEM  TAKES  PICTURE 
IF  s.  blob,  count  .>  0  THEN CHECK  FOR  BLOBS  IN  FQV 
CAi_i_  u  .  r  e  c  osr.  i  z  e  J  IDENTIFY  BLOBS 
x  =  s. proto. id 

CALl  u. where. are. you. proto;  PROVIDES  OBJECT'S  CENTROID 
«cer.  t  =  s.K.scale*s.xcenter.blobEx] 

TYPE  /Cl 

TYPE  "THE  X  COORDINATE  OF  THE  CENTROID  (in  wws )  IS",  /D, 
/cent  =  s.K. scalers. ycenter.blooCx] 

TYPE  "THE  Y  COORDINATE  OF  THE  CENTROID  (in  rums)  IS",  /P, 
orien  =  180*s.onent.blobCx3/PI 
TYPE  "GRIENTATION  IN  DEGREES  IS",  /D,  orien 
T  :'DE  /CZ 

CALL  camera. center. offset 

SET"  spot  =  c  amera  1  I  TRANS  (  x  c  en  t ,  ycent,  0,  orien,  -20,  0) 


Z5  ; 

ZG 

test  =  INRANGE(sPct) 

27 

IF  test  0  H E N 

ZS 

t  Y - E  / 3 ,  /Cl,  " 

=  - =  WARN  I NG  <  =  =  =  "  , 

/  C  1 

ZS 

TYPE  "THE  OEJECT 

IS  OUT  OF  THE  ROBOT'S 

REACH" 

30 

TYPE  "  PLEASE  MO 

VE  THE  OBJECT  AND  TRY 

AGAIN  " 

T  i 

PI^RMPT  "  r  =  t  s  r  -  '  • 

p p r  c q  /PFTIIP^  Tfl  P n hj T 

I  N  U  E  •  “ 

3Z 

GOTO  10 

3-3 

E_SE 

34 

SPEED  100 

35 

MOVE  spot 

3b 

END 

3  7 

r~ 

Jw 

TYPE  /B,  /CZ 

32 

TYPE  /B,  "TnERE  ARE 

NO  BLOBS  IN  THE  CAMERA 

'S  VIEW 

40 

TYPE  /CZ 

41 

T v P E  "PLEASE  PLACE  AN  OBJECT  ON  THE  TABlE 

43 

TYPE  "AND  RERUN  ThE 

PROGRAM " ,  /CZ 

43 

END 

44 

RETURN 

END 

/Cl 


x  c  e  n  t 
y  cent 


4 
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p1  track. tars 
PROGRAM  track. tars 
1  ; 


3 

4 

5 
G 

7 

8 

Cl 


This  prcsram  exhibits  the  charateristics  of  a  Static  look-and- 
wove  visual  servo  system  during  vision-robot  system  (ORS7)  motion. 

Before  executins  this  p r  o  s  r a  m  ,  ensure  that  there  aren't  any 
prototypes  in  the  vision  system's  memory.  If  there  are*  type 
EX  U.BGOT,  before  executins  this  proaram.  this  is  necessary 
since  this  proaraie  will  track  any  tarset  placed  in  the  camera's 
FOO. 


1 0 
l  i 
12 
1  3 
14 
1? 
1G 
1  7 
18 
18 
20 


The  p r o s r a m  instructs  the  vision  system  to  take  a  Picture  and 
provide  the  x  any  v  offsets  of  the  centroid  of  any  tarset 
in  t  re  camera's  F  0  0 . 

Next  a  prcsram  entitled,  center,  is  called.  Center 

provides  the  neccessary  vision  information  to  provide  the  x  and  v 
cert ~oid  positions  for  correction  to  the  robot's  position. 

7ie  DO  loo®  will  repeat  itself  until  the  tarset  is  centered  in  the 
camera's  FOO,  and  the  arm  has  stopped. 


23 

24 

or 

27 

28 
28 
30 


5  SPEED  100 

M  2  v  E  ffstart. position 
BREAK 

C  A  u.  ■_  u  .  i  n  1 1  v  l  s  i  o  n  J  initialize  communication. 

u.numbiob  =  1 ‘  Bleb  i  will  o  e  compared  to  all  prototypes. 


Creek  to  make  sure  the  call  is  in  the  camera's  FOO. 


32 

32 


4G 

47 

4e 

49 


10  CA-_L  u  .  take  .  pi  cture 

IF  s. blob. count  7  1  THEN 


34 

Type 

/B 

3^ 

TYPE 

"  * 

2S 

TYPE 

"  <r 

37 

TYPE 

M  * 

38 

T  y  P  £ 

"  S* 

2S 

TYPE 

"  * 

40 

t  y  p  ~ 

"  * 

41 

type 

/  E 

42 

t  y  p  e 

" 

43 

TYPE 

/B 

44 

PROMPT 

45 

GOTO 

10 

THE  BAl_  IN  NOT  IN  THE  CAMERA'S 
FIELD  OF  MIEN  (FOO) 


/B,  /Cl 


■»  " 
•»  " 

# " 


=>  PLEASE  POSITION  THE  BAwL  IN  THE  CAMERA'S 
:  =  •?  "•  PRE55  < RETURN  >  TO  CONTINUE  <  =  =  =  =  =  " 


ELSE 


20 


HERE  current?  define  current  as  present  location. 
DO 

TIMER  1  =  0 


50 

51 

52 

53 

54 

55 

56 

57 

58 
53 
GO 

END 


END 


u.new.pict u  re  =  1 

u.fast  =  0 

CALL  u. where. are. 

r 

TYPE  "The  time  to 
CALL  center;  This 


you. proto,'  take  Picture  and  set 
x ,  y  centroids 

set  the  vision  info  was",  TIMER(l) 
pros ram  instructs  the 
robot  to  center  the  camera 
over  the  tarset. 


UNTIL  ball  ==  0 
GOTO  20 


FOO" 
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Pi  center 
PROGRAM  center 

1  ; 

2  1  THIS  PROGRAM  CHECKS  TO  SEE  IF  THE  CENTROID  OF  THE 

3  ;  BAt-L  IS  IN  THE  CENTER  OF  THE  CAMERA'S  FOO. 

A  ; 


5 

bail  =  1 

G 

xcen  =  s 

. k.scale*s. xcen ter. blobll 

7 

y cen  =  - 

s. K . sealers. 

y center. Dlobtl 

e 

TYPE  ->( 

POS  IS 

xcen 

g 

TYPE  "Y 

POS  IS  ->"  , 

y  cen 

10 

TYPE  /Cl 

1  1 

IF  <  (ABS<  xcer.  }  <  10) 

AND  ( ABS ( y cen) 

•i  <■") 

ball 

=  0 

1  2 

ELSE 

i  4 

FOR 

i  n  to  id 

1 5 

SET  current 

=  SH I  FT ( c  urren t 

IS 

MOVE  current 

17 

END 

IS 

END 

1  9 

RETURN 

END 

10))  THEN 


B Y  k c  e n /IQ , 


c  e  n  /  15 
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p1  tracK.tara.scat 
PROGRAM  track . targ . scat 
1  ; 

2  ,  This  program  exhibits  the  charateristics  of  a  Static  looK-and- 

3  '  wove  visual  servo  system  during  vision-robot  system  (VRS>)  motion. 

4  • 

?  ;  The  program  instructs  the  vision  system  to  taKe  a  picture  a..d 
G  ;  provide  the  x  any  v  offsets  of  the  centroid  of  any  recognized  balls 
7  J  in  the  camera's  FOV. 

e  ; 

5  I  Once  the  call  has  been  identified  bv  the  system  program,  u. recognize, 

10  ;  a  program  entitled,  center. scat,  is  called.  Center. scat 

11  ‘  provides  the  neccessary  vision  information  to  provide  the  x  and  v 

12  ;  centroid  positions  for  correction  to  the  robot's  position. 

13  ' 

14  J  z  position  is  determined  by  the  size  of  the  ball  recognized.  The 

15  *  larger  the  ball,  the  closer  the  robot  is  to  its  desired  objective. 

ig  ; 

17  ;  The  DO  loop  will  repeat  itself  until  the  ball  is  centered  in  the 

18  ;  camera's  FOV,  and  the  arm  has  stopped  at  a  predetermined  height 

19  ‘  above  the  bail. 

20  ; 

21  J  See  the  program,  search . tracK »  for  additional  comments 

22  I  pertaining  to  the  general  use  of  this  program. 

o  o  * 
i*  J  t 

24  5  SPEED  100 

25  MOVE  #s tart . pcs l t i on 

2G  BREAK 

27  CALL  u.i  nit  vision,"  initialize  communication. 

28  u.numblob  =  11  Blob  1  will  be  compared  to  all  prototypes. 


Cr ecK  to  maKe  sure  the  ball  is  in  the  camera's  FOV . 

10  CAlL  u.taKe.Picture 

IF  s.  blob. count  <  1  THEN 

TYPE  /B,  /Cl  ***********„ 

TYPE  "*  *" 

TYPE;  the  BALL  IN  NOT  IN  THE  CAMERA'S  *" 

TYPE  “*  FIELD  OF  VIEW  (FOV)  *" 

TYPE  "*  *" 

TYPE  "#**»*»1MMHHH>*****#***#*#***##*#*#****if**" 

TYPE  /B,  /Cl 

TYPE  "  ==-===>  PLEASE  POSITION  THE  BALL  IN  THE  CAMERA'S  FOV 

TYPE  /E,  /Cl 

PROMPT  "  =====>  PRESS  <RETURN>  TO-  CONTINUE  <=====  " 

GOTO  10 

ELSE 

HERE  current;  define  current  as  present  location. 

TIMER  2=0 
DO 

TIMER  1  =  0 
BREAK 

u. new. picture  =  1 
u .  f  a  s  t  =  0 

CALL  u .where . are . you . proto J  take  picture  and  set 

J  x,  y  centroids 

CALL  u. recognize?  Identify  the  ball. 

id  =  s. proto. idJ  which  ball  was  identified. 
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>• 


bi  ; 
62  ; 
S3  ; 


64  ; 

65  TYPE  /Br  "#***#**#*#*#***#*#*###■**#*#***###*■**##*#**#**#** 

66  TYPE  "The  ball  recoanized  is  prototype  #"r  id 

67  TYPE  "The  time  to  aet  the  uision  info  wa*"»  TIMER(l)r  "  st 

66  CALL  center. scat!  This  prosram  instructs  the 

69  ;  robot  to  move  closer  and 

70  ;  centered  on  the  ball. 

71  UNTIL  ball  ==  0 

72  TYPE  TIMER ( 2 ) 

73  END 

74  PROMPT  "■«««■*>  PRESS  < RETURN >  TO  RUN  THE  PROGRAM  AGAIN  <=====" 

75  GOTO  5 

76  RETURN 
.END 
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Pi  centsr.seat 
PROGRAM  center. scat 
i  ; 


3 

4 

5 
G 

7 

8 
9 

10 
1  1 
12 

13 

14 
1? 
16 

17 

18 
19 
2'  O 


This  program  checks  to  see  if  the  centroid  of  the  ball  is  in 
the  center  of  the  camera's  FCV.  Also,  depending  to  which 
prototype  was  identified,  the  camera  will  move  down  in  the 
~z  airection  an  appropriate,  predetermined,  amount. 


The  x  and  y  locations  (s.xcenter.blobCD  and  s.v center. blobCl) 
provided  by  u. where. are.  you. proto  are  scaled  by  the  scale 
factor,  s . k . scale.  s . k . scale  is  determined  when  the  program  u. 
u.k. teach  is  executed.  See  the  program,  search . center ,  for  a 
further  explaination  of  s.k. scale. 


xcen  and  ycen  are  the  resultant  values  when  the  x  and  y  locations 
are  scaled.  It  should  be  noted  that  ycen  is  the  negative  value 
of  s.y center. falobC].  Tnis  occurs  since  the  robot's  x,y  frame  is 
different  than  the  camera's  x»y  frame. 

It  should  also  be  noted  that,  ycen,  when  scaled  by  s.k. scale 
does  not  provide  accurate  results.  Through  experimentation  it 
was  found  that  if  ycen  is  multiplied  by  2/3's,  good  results  occur 


24 

25 
2G 
27 
29 

29 

30 

31 


The  ball  will  be  considered  'centered'  if  the  centroid  is  within 
'+  or  -'  10  mm.  Once  the  object  is  identified  as  a  prototype 
greater  than  #G.  the  robot  will  not  move  any  closer  to  the  ball. 

Tne  FOR  Icop  in  this  program  breaks  up  the  desired  robot  motion 
into  only  10  steps.  This  allows  smooth  movement  with  minimal 
overshoot  of  the  target.  As  can  be  seen  in  the  loop,  depending 
on  which  ball  is  identified,  the  arm  will  move  down  a  designated 
amount.  Since  the  prototypes  were  trained  at  25  mm  increments 
the  value  of  downward  movements  should  be  factors  of  25. 


32  SPEED  50  ALWAYS 


33  ball  -  1 

34  xcen  =  s.k.scale*s.xcenter.blobCidl 

35  ycen  =  -s.K.scale*s. ycen ter. blobEidl 

3G  TYPE  "X  POS  IS  xcen 

37  TYPE  "Y  POS  IS  ycen 

3B  TYPE  "**##**#********♦***♦*«*********#*»##******###«#*#*"  , 


39 

40 
4  1 

42 

43 

44 

45 
4G 

47 

48 

49 

50 

51 

52 

53 

54 

55 
5G 

57 

58 

59 

60 


I  Chech  tc  see  if  the  ball  is  centered  and  close  enough  to  the  ball 
■ 

J 

IF  ((ASS (xcen)  <  10)  AND  (ABS(vcen)  <  10)  AND  ((id  <  1)  OR 
ball  =  0 

ELSE 

FOR  i  =  1  TO  10 

CASE  INT(id)  OF 
VALUE  0,  91 
zoom  =  0 
VALUE  1,  2,  31 
zoom  =  -150 
VALUE  41 

zoom  =  -75 
VALUE  5,  7*. 

zoom  =  -50 
VALUE  G,  81 
zoom  =  -25 

END 
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/Cl 


(id  > 


62  ; 

63  ; 

64  ; 

65  J  The  next  IF  statement  accounts  For  the  initial  offset  between  the 

66  !  position  of  the  camera  mount  with  respect  to  the  PUMA  end  effector. 

67  ’  See  search. center  For  an  explanation. 

66  ; 

69  IF  ((id  <>  0)  AND  (id  <  4))  THEN 

70  SET  current  =  SHIFT( current  BY  xcen/10*  ycen/15+3 

71  ELSE 

72  SET  current  =  SHIFT( current  BY  xcen/10r  ycen/15» 

73  END 

74  END 

75  HOME  current 

76  END 

77  RETURN 
.END 


.  5  t  £ 
zoom/ 
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Pi  search 
PROGRAM  search 


his  proaram  will  so  through  a  "square  conical"  search 
(going  from  the  largest  possible  area,  to  the  smallest) 
in  an  attempt  to  acquire  a  target  (ball)  in  the  search 

Once  the  ball  has  been  acquired  bv  the  vision  system, 
a  program  entitled,  search . tracK .  is  called.  This 
program  will  instruct  the  robot  to  center  the  ball  in  t 
camera's  FOU  and  lower  the  camera  closer  to  the  ball. 


area . 


If  the  ball  is  not  acquired  during  the  first  search,  the 
search  will  be  accomplished  over  and  over,  until  a  ball  is 
placed  in  the  search  area,  or  the  user  terminates  the  program. 


Once  the  ball  has  been  aqcuired.  the  syste 
continuous  "tracking"  mode.  In  this  mode, 
the  bail  anywhere  in  the  camera's  FQ1.1.  and 
system  will  trad;  the  ball  (the  roDct  will 
the  ball.  Just  track  it). 


system  will 
mode .  the  u 


will  so  into  a 
the  user  can  move 
the  robot/uision 
not  move  closer  to 


If  the  Dali  is  removed  completely  from  the  camera' 
search  will  start  over  from  the  beginning.  The  se 
starting  point.  #search. start,  is  defined  as  a  UAL 
(refer  to  a  UAL  II  User's  Guide  for  a  definition  o 


camera's  FOU,  the 
The  search 's 
s  a  UAL  II  precision 
ition  of  precision  point) 


SpEED  100 

MOVE  #search . s ta-t ' 


predifined  starting  point 


count  =  Cl  Initialise  counter  used  to  instruct  the 
J  program  to  accomplish  tracking  only. 
ti>  =  01  Initialize  counter  for  square  conical  search. 
BREAK!  Next  command  waits  until  robot  motion  stops. 

speed  so  always 

m  -  m*i 

CALL  u . in i t v i s i on ;  Initialize  vision/rcbot  communication 
u.numbicb  =  1J  Blob  1  will  be  compared  to  all  prototypes. 


HERE  start 

CALL  u . take . p i c t ure ;  System 


program  which  checks  for  blobs. 


foil  owing 


aqcuisition 


IF-THEN-ELSE  i 
n  of  the  ball. 


conduct 


square  search 


search  will  continue  until,  u . take . picture,  finds 


with  an  area  greater  than 
in  the  camera's  FOO. 


the  minimum  blob  pixels  (set 


blob 

by  user) 


s. blob. count  <  1  THEN 
CASE  INT(m)  OF 

UALUE  1.  2.  3.  12, 
MOOE  SHIFT ( start 
UALUE  4,  5.  6.  141 
MOVE  SHIFT (start 
UALUE  7,  B.  9.  151 
MOUE  SHIFT ( start 

ualue  io,  u,  is: 

MOUE  SHIFT< start 
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BY  BO) 


-80) 


-S5 ) 


62 

I 

63 

• 

t 

64 

• 

t 

65 

ANY 

66 

TYPE 

/B 

67 

TYPE 

"SEARCH  HAS  BEEN  COMPLETED,  THE  BALL 

WAS" 

se 

TYPE 

"NOT  IN  THE  SEARCH  AREA.",  /Cl 

9 

TYPE 

"THE  SEARCH  WILL  NOW  REPEAT  ITSELF.", 

/B 

70 

t 

71 

TYPE 

"PLEASE  PLACE  THE  BALL  IN  THE  SEARCH 

AREA,  " 

72 

TYPE 

"OR  TERMINATE  THE  PROGRAM  BY  TYPING,' 

,  /Cl 

73 

TYPE 

"  'A'  FOLLOWING  BY  PRESSING  THE  < RETURN! 

74 

; 

75 

DELA 

Y  2;  Delay  proaram  execution  by  2  seconds. 

76 

GOTO 

50 

77 

END 

78 

BREAK 

79 

ELSE 

80 

• 

r 

81 

• 

f 

The  follow  ins  DC  loop  will  continue  forever,  unless  the 

user 

62 

• 

i 

wishes  to  terminate.  Th  terminate  prosraw  execution,  p 

res  s 

83 

t 

"A"  followed  bv  < RETURN! . 

84 

a 

1 

85 

1 

If  the  ball  is  removed  from  the  camera's  FOv.  The  search  wl. 

96 

t 

start  over. 

KEY"  r 


87  ; 

88 

89 

90 

91 

92 

93 
14 
35 

9E  ; 

97 

98 

99 
100 
101 
102 

1 03 

104  ; 

105 
10G 

107 

108 

109 

110 
1 1 1 

112  : 

1 13 

114 
1  15 


DO 

CA:_L  •’search .  tracK 
IF  lost  <>  0  THEN 

"THE  BALl.  HAS  BEEN  ACQUIRED"  ,  /Cl 
"TO  STOP  THIS  program:",  /Cl 

TYPE  'A'  FOLLOWED  BY  PRESSING  THE  " 
< RETURN >  KEY.",  /Cl 

END 


TYPE  /B, 
TYPE  /B, 
TYPE  " 
TYPE  " 


count  = 


CAi_L  u. 


15J  Reset  counter.  This  is  to  prevent 
;  the  robot  from  m  o  v  i  n  a  closer  to  the 
ball.  However  tracKina  can  still  be 
J  accomplished. 

tahe . pivture ;  ChecK  to  see  if  ball  is 
;  is  still  in  the  camera's  FOO. 


UNTIL  s. bloc .count  ==  0 


TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 

TYPE 


zr  i  ..********»************************************ 

»*  THE  BALL  HAS  BEEN  REMOVED  FROM  *" 

THE  CAMERA'S  FOO,  THE  SEARCH  WILL  *" 

"<r  REPEAT  UNTIL  THE  BALL  IS  REACQUIRED  *" 

*  " 

11  £.  # 
«**#******************#***********************’'  *  /C 


GOTO  50 

END 

GOTO  5 


.END 


/Cl 
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p1  search. tracK 
PROGRAM  search. tracK 
1  ; 


.  3 

4 

5 
G 

7 

8 
9 

10 

11 

12 

13 

14 

15 
1G 

17 

18 

19 

20 

21 
'■y  o 

22 

24 

25 
2G 
27 

'8 

29 

30 

31 

32 

33 

34 

35 
3G 

37 

38 

39 

40 

41 

42 

43 

44 

45 
4G 

47 

48 

49 

50 

51 

52 

53 
.4 

55 

5G 

57 

58 

59 
GO 


This  program  instructs  the  vision  system  to  taKe  a  picture 
and  ai v e  the  x  and  y  offsets  of  the  centroid  of  any  reco3nized 
balls  in  the  camera's  FOU. 

Once  the  ball  has  been  identified  bv  the  system  program* 
u. recoanize*  a  program  entitled*  search. center*  is  called. 

Search . center  provides  the  necessary  vision  information 
to  provide  the  x  and  y  centroid  positions  for  correction  to 
the  robot's  position. 

z  position  is  determined  by  the  size  of  the  ball  recognized. 

The  larger  the  ball*  the  closer  the  robot  is  to  its  desired 
objective. 

The  DO  loop  will  repeat  itself  until  the  ball  is  centered  in 
the  camera's  FOU,  and  the  arm  has  stopped  at  a  predetermined 
height  above  the  ball. 

NOTE:  The  reason  the  robot  is  not  lowered  all  the  way  to  t.he 

ball  is  that  if  the  ball  encompasses  a  majority  of  the  camera's 
FOU »  the  cevitrcid  position  can  not  be  calculated  accurately*  and 
the  robct/vision  system  will  overshoot  back  and  forth  for  an 
extended  period  of  time. 

NOTE!  The  different  ball  sizes  are  trained  to  the  vision  system 
ahead  of  time.  Using  a  ball  alleviates  the  problem  of  not  having  an 
autofocus  camera  lens.  Even  though  the  ball  sets  blurry  as  the 
camera  sets  closer*  it  will  still  remain  circular  in  the  camera's 
2- dimensional  FOU .  u . train . blob  was  used  to  train  the  vision  system. 
Training  occurs  as  follows: 

1)  The  user  defines  a  starting  position  (the  one  used  here  is 
designated  as  the  following  presion  point*  #s tart . pos i t i on ) . 

2)  u. train. blob  is  called*  the  first  object  trained  will  be 
designated  as  prototype  '1'. 

3)  Once  training  has  been  accomplished*  the  user  should  verify 
the  training  was  succesful.  This  is  accomplished  by  using 
the  light  pen  to  select  the  o  PROTOTYPE  MENU*  and  then 
checkins  the  o  DISPLAY  DISCRIMATIDN  MATRIX.  There  should  be 
a  large  blocK  next  to  the  CURRENT  SET  row.  If  not*  the  ball 
should  be  retrained. 

To  mane  life  easier,  the  balls  have  been  already  trained  at  25  mm 
intervals  and  stored  on  tape.  The  tape  is  entitled  'VISION  SERUO 
CONTROL.  TARGETS'.  These  can  be  loaded  directly,  without  any  additional 
training. 

Tne  timer  is  used  to  delay  program  execution  of  the  prosram 
and  display  the  amount  of  time  required  for  the  vision  system 
to  process  the  imase  and  send  the  required  data  bacK  to  the 
robot  processor. 

Since  this  particular  prosram  is  worKins  as  a  Static  look-and- 
move  system*  the  delay  is  required  to  ensure  the  robot  has 
stopped  movins  before  the  next  Picture  is  taKen.  The  reason  for 
this  is!  if  the  vision  system  takes  a  picture  in-between  trained 
ball  sizes*  the  Picture  taken  will  not  be  an  identifiable  object. 

(The  area  of  the  ball  will  not  match  any  area  trained  earlier) 

Thus  the  vision  system  will  not  know  how  far  down  to  move  the  arm. 
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62 

62 

G4 

65 

66 
67 

-69 

9 

^70 

71 

72 

73 

74 

75 

76 

77 

78 

79 
60 
81 
62 

83 

84 

85 
BG 

87 

88 
89 
SO 

91 

92 

93 
'4 
95 

.  END 


SPEED  25 

HERE  current?  Define  current  as  present  location. 

The  Followins  2  parameters  initialized  are  used  incase  the  vision 
sv stem  sets  lost*  ie.  it  thinKs  it  has  the  ball  in  its  F00»  but 
i n  real i  t  y  it  doesn't. 

xcen  =  0 
xcenl  =  1 

DO 

CALL  u . i n i t v i s i on J  Initialize  communication. 

TIMER  1  =  0 
BREAK 

u. new. Picture  =  1 
u .  f  a  s  t  =  0 

CALL  u . where . are . yo u . pro  to J  take  Picture  and  aet 

;  Xi  •/  centroids 

CALL  u.recosnizeJ  Identify  the  ball. 

id  =  s. proto. idJ  This  comes  form  u.recoanize. 

;  It  identifies  which  ball  was  identified. 

TYPE  /Bi  "*********************************** ****** ********** " 

TYPE  "Tne  ball  recoanized  is  prototype  #"»  id 

TYPE  "The  time  to  aet  the  vision  info  was" >  TIMER(1>»  /Cl 

CA:_L  search. center;  This  proaram  instructs  the  robot 
J  to  move  closer  and  centered  on  the  ball. 

UNTIL  ball  ==  0 

TYPE  " T I  ME  IS  ",  TIMER(ll) 

RETURN 
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•  p1  search . center 
•PROGRAM  search . center 
i  ; 


5 

G 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
r>r> 

23 

24 

25 

26 
27 
29 


This  proaram  checKs  to  see  if  the  centroid  of  the  ball 
is  in  the  center  of  the  camera's  FOO.  Alsor  depending  on 
which  prototype  was  identified/  the  camera  will  move  down 
in  the  '-z  '  direction  an  appropriate/  predetermined/  amount. 

The  x  and  y  locations  ( s . xc en t er . b  1  ob C 3  and  s . yc en t er . b 1 ob C 3 ) 
provided  by  u . where . are . you . pro  to  are  scaled  by  the  scale 
factor/  s . K . scale.  s . K . scale  is  determined  when  the  program 
u . K . teach  is  executed.  It  is  important  to  get  an  accurate  scale 
factor/  with  the  camera  mounted  to  the  robot.  The  following 
should  be  accomplished  to  ensure  this! 

1)  Put  the  arm  in  the  starting  position  ( #start . pcs i t i on ) . 

2-  Execute  u.K. teach,  maKe  sure  the  disK  will  be  in  the  camera's 
FOU  when  the  robot  is  moved  out  of  the  way. 

The  value  cf  s.K. scale  should  be  approximately  .5. 

xcen  a  no  v'cen  are  the  resultant  values  when  the  x  and  y  locations 
are  scaled.  It  should  be  noted  that  vcen  is  the  negative  value 
of  s.v center. blob£3.  This  occurs  since  the  robot's  x,y  frame  is 
different  than  the  camera's  x,y  frame. 

It  should  also  be  noted  that,  ycen,  when  scaled  by 

s.K. scale  does  net  provide  accurate  results.  Through  experimentation 
it  was  found  that  if  ycen  is  multiplied  by  2/3's,  good  results  occur. 

The  ball  will  be  considered  centered  if  the  centroid  is  within  '+  or 
10  mm.  Once  the  object  is  identified  as  a  prototype  9 
the  robot  will  not  move  any  closer  to  the  ball. 


9 

wO 

31 

32 

33 

34 

35 
3G 
37 
3e 

39 

40 

41 

42 

43 

44 

45 
4G 

47 

48 

49 

50 

51 

52 

53 
^4 

5 

58 

57 

58 


;  The  FOR  loop  is  used  to  breaK  up  the  robot  motion  into  10  steps. 

I  This  allows  smooth  movement  with  minimal  overshoot  of  the  target. 

;  As  can  be  seen  ir,  the  loop,  depending  on  which  ball  identified, 

;  the  arm  will  move  down  a  designated  amount.  Since  the  prototypes 
;  were  trained  at  25  mm  increments,  the  value  of  downward  movements 
J  should  be  factors  of  25. 

speed  50  always 

lost  "  1 
ball  =  1 

xcen  =  s.K. scalers. xcen  ter. blobtid3 
ycen  =  -s. K . s  c  a  1  e  *  s  . ycenter.blobiidD 

TYPE  "The  ball's  centroid  in  the  'X'  direction  is",  xcen 
TYPE  "The  ball's  centroid  in  the  'Y'  direction  is",  ycen 
TYPE  "*#***************************■**•»■************'*******"  ,  /Cl 

;  ChecK  to  see  if  the  vision  system  is  huns.  . 

IF  xcen  =  =  xcenl  THEN 

TYPE  /Cl,  "#*****#****##*##**#*#*#*#***#***#*****#*" 

TYPE  "*  *" 

TYPE  "* _  THE  BALL  HAS  BEEN  LOST  *" 

TYPE  "*  *" 

TYPE  "*****#*************#*******#******#*****",  /C2 
lost  =  0 
ball  *  0 

ELSE 

■ 

f 

;  ChecK  to  see  if  the  ball  is  centered  and  close  enough  to  the  ball. 


59  : 

60  ; 
si  ; 
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63 

64 

65 

66 

67 

68 
*  ~9 


73 

74 

75 

76 

77 

78 
73 
80 
81 
82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 
5 

jS 

97 

9e 

99 

100 

101 

102 

103 

104 

105 

106 

107 

108 

109 

110 
111 
112 

113 

114 

115 

116 

117 

118 
1 19 
<  ?0 

1 

1 22 

123 

124 

125 

126 
127 


IF  <<ABS(xcen)  <  10)  AND  (ABS(yeen)  <  10)  AND  (id  >  8))  THEN 
o  a  1 1  *  0 

ELSE 

count  *  count+i 
FOR  i  «  1  TO  10 

CASE  INT(id)  OF 

value  o: 

The  following  IF  statement  is  to  account  For  the  situation  when 
the  ball's  initial  position  in  the  search  area  is  at  a  distance 
greater  than  that  trained  to  the  uision  system.  The  search  area 
is  such  that  at  no  point  in  search  area  will  the  ball  be  at  a 
distance  greater  than  75  mm  from  an  initial  training  position. 

IF  count  <=  4  THEN 
200m  =  -25. 

ELSE 

2  c  0  01  =  0 

END 

VALUE  1.  2,  31 
zoom  ~  -150 
VALUE  41 
zoom  =  -75 
VALUE  5 >  I'. 
zoom  =  -50 

value  6,  a: 

zoom  '  -25 
VALUE  91 
zoom  *  0 

END 

The  following  IF  statement  is  used  once  the  ball  has  been 

acquired  and  zoomed  into.  The  vision/robot  system  will  Just 

become  a  tractor.  This  is  a  safety  measure*  because  in  the 

tracking  mode*  if  only  part  of  the  ball  is  in  the  camera's 

FOV  when  a  picture  is  taKen,  the  prototype  identified  might  not 

be  what  the  actual  prototype  is.  For  example*  if  the  blob  identified 

was  new  prototype  #1*  the  robot  would  thinK  that  it  should  lower 

itself  150  mm*  thus  putting  itself  out  of  its  possible  range  of 

motion. 

IF  count  >  B  THEN 

SET  current  =  SHIFTCcurrent  BY  xcen/iG*  ycen/25) 
IF  ( <  ABS  <  xcen )  <  10)  AND  (ABS(ycen)  <  10) )  THEN 
ball  =  0 

END 

GOTO  25 

END 

The  next  IF  statement  accounts  for  the  initial  offset  between 
the  position  of  the  camera  mount  with  respect  to  the  PUMA 
end  effector.  This  is  a  problem  because  x*y*z  coordinates  in 
the  VAL  II  controller  are  based  on  the  location  of  Joint  6* 
not  Joint  3  (where  the  camera  is  presently  mounted). 
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129  ; 

130 

IF  < ( id  <>  0)  AND  (id  <  3))  THEN 

131 

IF  m  =»  2  THEN 

132 

SET  current 

*  SHIFT( current 

BY 

Keen/ 10+1 »  ycen ) 

133 

END 

134 

IF  <<m  **  3)  OR 

(m  «*  4) )  THEN 

-*  35 

SET  current 

*  SHIFT< current 

BY 

xcen/10+2»  ycen) 

i  ,G 

ELSE 

*1 37 

SET  current 

=  SHIFT( current 

BY 

xcen/10»  ycen/1) 

138 

END 

139  ; 

140  ; 

141 

ELSE 

142 

IF  id  -=  3  THEN 

143 

SET  current 

=  SHIFT( current 

BY 

xcen/10»  ycen/1) 

144 

ELSE 

145 

SET  current 

=  SHIFT( current 

BY 

xcen/10r  ycen/1) 

14G 

END 

147 

END 

148 

^  cr 

MOVE  c urren t 

149 

END 

150 

END 

151 

END 

i52 

xceril  * 

xcenl  This  sets  up  the 

checK  for  the  situation  where 

153 

‘  the  uision  s  v  stew  had  the  faallr  then 

lost  it. 

154 

RETURN 

.END 

Task  2:  Dynamic  Visual  Servo  Control  Program  Listings 


p1  tract; .  tara.  dcat 
PROGRAM  tract; .  tars  .  dcat 
1  : 


3 

"  4 
5 
G 
7 

e 

9 

10 
1 1 
1Z 

13 

14 


This  proaram  exhibits  the  c hara t er i s t  i  c s  of  a  Dynamic  looK-and- 
iwove  visual  servo  system  when  the  arm  moves  in  a  downward  direction. 
The  remainins  movements  are  done  in  the  more  classical  Static  looK- 
and-move  servoina  method. 

The  proaram  instructs  the  vision  system  to  taKe  a  picture  and 
provide  the  x  any  v  offsets  of  the  centroid  of  any  recosnized  balls 
in  the  camera's  FOV. 

Once  the  ball  has  been  identified  bv  the  system  proaram,  u.recoanize 
a  proaram  entitled,  center. dcat,  is  called.  Center. dcat 
provides  the  neccessary  vision  information  to  provide  the  x  and  v 
centroid  positions  for  correction  to  the  robot's  psoiticn. 


15 
1G 
17 
IS 

19 

20 

9  < 

«£  at 

In 

23 

24 

25 
2G 
27 

*8 

29 

30 

31  : 

32  ; 
32  : 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44  . 

45 
4G 

47 

48 

49 

50 

51 

52 

53 
i4 

55 

56 

57 

58 

59 
80 


z  position  is  determined  bv  the  size  of  the  ball  recosnized.  The 
larser  the  ball,  the  closer  the  robot  is  to  its  desired  objective. 

The  DO  leap  will  repeat  itself  until  the  ball  is  centered  in  the 
camera’s  FGO ,  and  the  arm  has  stopped  at  a  predetermined  heisht 

above  the  ball. 

See  the  Proaram,  search. ball . track,  for  additional  comments 
pertaining  to  the  general  use  of  tms  program. 

5  SPEED  100 

MOVE  #start . pos ition 

g  PJ  C  £  (•' 

E„„  to  ..He  tori  the  ball  is  in  the  camera's  FOV. 

10  CA;_L  u.  take. Picture 

IF  s.  blob,  count  <  1  THEN 

XYpi  "*****i**********************************" 

«*  THE  BALL  IN  NOT  IN  THE  CAMERA'S  *" 

TYPE  "*  FIELD  OF  VIEW  <FDV> 

TYPE  "* 

TYPE  “■it■^^***■*****'IM,'*' 

IJpl  PLEASE  POSITION  THE  BALL  IN  THE  CAMERA'S 

IZan/j* ■’  >  PITESS  <RETURN>  TO  CONTINUE  <•  —  -  “ 

GOTG  10 

ELSEHERE  current;  define  current  as  present  location. 

TIMER  2=0 
DO 

TIMER  1  =  0 
u. new. Picture  =  1 

^r;.;here. are. rewrote;  JO. 
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FOV" 


62  ; 

63  ; 

64 

65  ; 

66  ; 


67  TYPE  /Br  "*###+#*#*#*«*******###*#*****####*#####**#*###♦#« 

76  TYPE  "The  ball  recognised  is  prototype  #"»  id 

3  TYPE  "The  time  to  set  the  vision  info  was"»  TIMER(l).  "  sel 

70  CALL  center,  dcat?  This  program  instructs  the 

71  ;  robot  to  move  closer  and 

72  :  centered  on  the  ball. 

73  UNTIL  ball  ==  0 

74  TYPE  T I MER ( 2 ) 

75  END 

76  PROMPT  "  =  =  =  =  =  =  ;•  PRESS  < RETURN >  TCI  RUN  THE  PROGRAM  AGAIN  <«***=" 

77  GDTO  5 

76  RETURN 

END 


F' center. d cat 
PROGRAM  center. dcat 


3 

4 

5 
G 

7 

8 
g 

10 
1  1 
i  r> 

13 

14 

15 
1G 

17 

18 

19 

20 
r  i 


24 

25 
2G 

*“■»  y 

e 

22 

30 

31 

32 

33 

34 

35 
3G 

37 

38 

39 

40 

41 

42 

43 

44 

45 
4G 

47 

48 

49 

50 

51 

52 

53 
34 
55 
5G 

57 

58 
53 
GO 


This  program 
the  center 
Prototype 


checks  to  see 
of  the  camera 


if  the  centroid  of  the  ball 
s  FGV.  Also,  depending 
was  identified,  the 
direction  an 


camera  will 


The  x  and  r 

provided  by 
factor.  s.K, 


appropriate,  predetermined, 
(s. xcen ter. blobCl 


to 

move  down  i  r. 
amount . 


1  5 

which 
the 


l  r * 


'^x^^^-blobCl  and  s. -/center. blobCJ) 
u. where. are. you. proto  are  scaled  by  the  scale 
(1  ,  ,scale*  s.f..  scale  is  determined  when  the  program 

Purthlr^p  Xi  eKec',te'3-  See  proaram .  search .  center .  for  a 
further  exrlaination  of  s.K. scale. 


y  locations 


the  negative  val 


u  e 


-veer  and  ycen  are  the  resultant  values  when  the  x  and 
are  scaled.  It  should  be  noted  that  ycen  i( 

of  s . veenter . b lob C 3  .  This  occurs  since  the  rooot's  x . y  frame  is 
than  the  camera's  x.y  Frame. 

-  h  o  u 1 d  a  1  sc  be  noted  that.  ycen.  when  scaled  d  v  s.K. scale 
;  not  provide  accurate  results.  Through  experimentation  it 
was  found  that  if  ycen  is  multiplied  by  2/3  s 


different 
It  : 
doe1 


soo d  results  occur. 


'he  ball  will  be 
'+  or  10  mm. 
ire a ter  than  #8. 


considered  'centered'  if  the 
One e  the  object  is  identified 
the  r o  b  c  t  will  not  move 


an  y 


entroid  is  within 
as  a  prototype 
closer  to  the  ball. 


.s  proaram  breans  up  the  desired  robot  motion 
This  is  what  allows  the  dynamic  looK-ano-move 
occur.  As  the  arm  is  moving  down,  a  Picture  is 


Tr. e  FOR  locn  in  th: 
into  o  n 1 ,  2  steps, 
visual  tract;  ins  to 
tahen  as  soon  as  possible  in  program  execution.  The  commands  are 
serial,  but  a  form  of  parallel  operation  is  possible  because  once 
tre  MOVE  command  is  executed  tne  next  commands  are  addressed. 
Therefore  a  Picture  is  requested  while  the  arm  is  moving  form  its 
last  MOVE  command.  See  search . center  for  information  pertaining  to 
the  v a  1  u e  s  for  dowr.ua r  d  m o  v e  m e n  t . 

SPEED  25  AuWAYS 
ball  =  1 

xcen  =  s. K. s  c  a  1  e  *  s  . xcen  ter. olobCid  3 
ycen  =  -  s  .  K  .  s  c  a  1  e  *  s  .  •/  c  e  n  t  e  r .  b  1  o  b  C  :  d  2 
TYPE  ">C  POS  IS  xcen 

TYPE  "Y  POS  IS  ycen 

TYPE  "■iHHt^********'**-***#******-#**^*#**#*#*****##*#*****#*" 


/C 


CnecK  to  see  if  the  ball  is  centered  and  close  enough  to  the  ball 


IF 


(  ( A8S (xcen) 
ball  =  0 

ELSE 

FOR  l  =  1  TO  2 

CASE  I N T  < id)  OF 
VALUE  o: 

:oom  -  0 
VALUE  1,  2,  31 
zoom  =  -150 
VALUE  4,  5,  Gl 
zoom  =  -50 
VALUE  7,  8,  91 
zoom  3  0 

END 


<  10)  AND  ( AES ( ycen) 


10)  AND  ( ( id  <  1 )  OR  (id 
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T 


gz  ; 

G3  ; 

64  : 

65  : 

66  f  The  next  IF  statement  accounts  For  the  initial  offset  between  the 

67  J  position  of  the  camera  mount  with  respect  to  the  PUMA  end  effector. 
T8  J  See  search . center  for  an  explanation. 

j9  ; 

70  IF  ((id  <>  0)  AND  (id  <  4))  THEN 

71  SET  current  ■  SHIFT  ( current  BY  xcen/2.  ycen/4+25. 

72  ELSE 


72 

IF  id  ■* 

4  THEN 

74 

SET 

current 

=  SH1FT< current 

BY 

xcen/2.  ycen/4 

75 

ELSE 

76 

SET 

c  urr en  t 

=  5HIFT( current 

BY 

xcen/2.  ycen/4 

77 

END 

76  END 

79  MGME  current 

BO  END 

61  END 

82  RETURN 

•  END 


zoom ) 

15  r  > 
zoo) 


15G 


Appendix  C 

User 1 b  Manual 

Appendix  C  contains  the  User’s  Manual  for  setting  up 
this  particular  vision-robot  system  (VSR)  for  the  various 
tasks  accomplished  in  my  thesis.  The  Manual  provides  step- 
by-step  instructions  to  enable  the  user  to  execute  the 
various  programs. 


Directions  for  Operating  VRS 


Step _ Direction _ 

===>  NOTE  <=== 

It  is  assumed  that  the  user  has  read  the  following 
references  -->  (16;  17;  18),  before  using  the  VRS. 

1.  Mount  the  camera  to  the  camera  stand,  with  the 
camera  pointing  towards  the  black  part  of  the 
table,  when  looking  at  the  image  through  the 
camera . 

2.  Load  the  vision  system's  operating  software  by 
inserting  the  Univision  S/W  tape  into  the  "TAPE  0" 
slot  located  on  the  right  side  of  the  vision 
system’s  video  monitor.  Depress  the 
"RESTART/RELOAD"  button  (a  blue  light  should  be 
visible) . 

3.  Turn  the  key  to  the  "ON”  position.  The  LED  by  the 
tape  drive  will  flash  while  the  system  is  loading. 

===>  NOTE  <=== 

It  takes  approximately  5  minutes  to  load  the 
system.  Therefore,  the  user  may  accomplish  Steps 
16-26  to  bring  up  the  robot,  while  loading  occurs. 


A  correct  load  will  results  when  the  Machine 
Intelligence  Univision  (3.09-B)  logo  is  displayed. 

4.  Remove  the  tape  from  the  drive. 

5.  Press  the  light  pen  to  any  white  part  of  the 
screen . 

===>  NOTE  <=== 

The  following  steps  set  up  the  vision  system  for 
Subtask  1.  For  setting  up  the  vision  system  for 
the  remaining  Subtasks  and  Tasks  accomplish  Steps 
15. 

6.  Select:  o  SYSTEM  SETUP 

o  VIEW  CURRENT  CAMERA 

7.  Place  the  calibration  disk  in  the  camera’s  field  of 
view  ( FOV ) . 
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8.  Focus  the  camera  and  adjust  the  aperture  to  get  as 
clear  an  image  as  possible. 

9.  Press  the  light  pen  to  any  part  of  the  screen  and 
then  select  the  following: 

o  SET  THRESHOLD 

Set  the  threshold  value  to  128  by  pressing  the 
light  pen  to  an  arrow  until  128  shows  up  in  the 
upper  right  corner  of  the  monitor. 

128  provides  the  best  contrast  when  using  white 
objects  against  a  black  background. 

10.  Select:  o  QUIT 

o  OPERATING  OPTIONS 

Ensure  "ONLY"  the  following  options  are  "ON”. 

o  CONNECTIVITY  ANALYSIS 
o  FIRST  MOMENTS 
o  KEEP  ALL  BLOBS 
o  NOISE  SUPPRESSION 
o  OUTLINE  BLOBS 
o  PERIMETERS 
o  RECOGNITION 
o  SECOND  MOMENTS 

11.  Select:  o  QUIT 

o  QUIT 

o  EXPERT  OPERATION 
o  OPERATING  OPTIONS 

In  addition  to  the  options  listed  in  Step  10, 
ensure  "ONLY"  the  following  options  are  "ON". 

o  OBJECTS  ONLY  PROCESSED 
o  PROCESS  ALL  BLOBS 

===>  NOTE  <=== 

Other  options  are  possible,  refer  to  (17)  for  more 
information . 
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12. 

Select : 

o 

QUIT 

o 

QUIT 

o 

UNIVISION 

o 

VIEW  CURRENT  CAMERA 

13. 

READY  TO 

GO 

•  «  •  9  1  V 

14.  Ensure  the  robot  (operating  under  VAL  II)  is  up  and 
running.  If  not,  accomplish  Steps  16-26. 

15.  ONLY  the  following  options  from  the  various 
OPERATING  OPTIONS  should  be  selected  when 
accomplishing  Subtasks  2,  3,  4  and  Task  2. 

o  CONNECTIVITY  ANALYSIS 
o  FIRST  MOMENTS 
o  NOISE  SUPPRESSION 
o  OBJECTS  ONLY  PROCESSED 


=  =  =  >  NOTE  <=== 

The  following  steps  will  setup  the  PUMA  560  under  the  VAL  II 

operating  system. 

16.  Turn  of  the  robot’s  dumb  terminal. 

17.  Turn  the  power  "ON"  the  Unimate  Computer/Controller. 

18.  A  message  should  appear  asking  the  user  to  load  VAL  II 
from  floppy  disk. 

19.  Insert  the  5  &  1/4"  floppy  disk  entitled  "VAL  II 
560. 1. 4. B"  into  the  Unimate  disk  drive. 

20.  Type  "Y  <RETURN>"  to  the  prompt  asking  the  user  to  load 
VAL  II . 

21.  Enter  the  number  "798"  for  the  robot  serial  number. 

22.  Approximately  2  minutes  later,  type  "Y  < RETURN) "  to  the 
prompt  asking  the  user  to  INITIALIZE  'the  system.  A 
system  prompt  will  appear. 
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===>  NOTE  <=== 

Initializing  the  system  has  the  effect  of  erasing  all 
programs  stored  in  memory.  Therefore,  ensure  a  copy  of 
all  software  is  stored  on  disk. 

23.  Remove  the  Unimation  disk  and  insert  the  Univision  I 
system  program  disk  935H3  which  contains  the  interface 
software . 

24.  Enter  the  following: 

==>  LOAD  UNIVISION1 . PG 

11  s.*.*  files  and  15  u.*.*  files  should  be  loaded  and 
displayed . 

25.  Remove  the  Univision  disk  and  insert  the  Vision  Servo 
Control  disk  into  the  Unimate  disk  drive. 

26.  Enter  the  following: 

==>  LOAD  VI SION. SERVO. CONTROL 

The  19  programs  listed  in  Appendix  B  should  be  loaded. 

27.  To  execute  Subtask  1,  type  the  following: 

EX  VISION. DEMO 

and  accomplish  the  OPTIONS  in  order  to  initialize 
communications,  calibrate  the  systems,  train  targets, 
and  execute  open  loop,  static  look-and-move  control. 

28.  For  the  remaining  Subtasks  and  Task,  the  camera  must  be 
mounted  on  the  third  joint  of  the  PUMA  560  (see  Figure 
10  in  Chapter  III)  and  the  table  removed  from  the  robot 
workspace.  Put  the  black  part,  of  the  table  on  the 
floor  for  a  work  area.  The  vision  system  must  also  be 
re-calibrated  by  executing  the  following  self- 
explanatory  program: 

EX  U.K. TEACH 

# 

which  determines  the  camera  scale  factor  (s.K. scale 
should  be  approximately  equal  to  0.5). 
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29.  To  execute  Subtask  2,  delete  all  prototypes  from  the 
vision  system’s  memory  by  typing  the  following: 

EX  U . BOOT 

Next  place  a  target  under  the  camera’s  FOY .  The  system 
is  ready  for  closed  loop,  static  look-and-move  visual 
servo  control.  Enter-  the  following: 

EX  TRACK. TARG 

30.  Before  executing  Subtask  3,  the  trained  targets  must  be 
loaded  into  the  vision  system’s  memory.  This  is 
accomplished  by  inserting  the  vision  cassette,  labeled 
"VISION  SERVO  CONTROL  TARGETS",  into  the  vision 
system’s  "Tape  0"  slot.  Next  select  the  following 
with  the  light  pen: 

o  PROTOTYPE  MENU 

o  LOAD  ALL  PROTOTYPES  FROM  MEMORY 

The  system  should  now  be  ready.  Place  the  white  ball 
below  the  camera  and  enter  the  following: 

EX  TRACK. TARG. SCAT 

31.  To  execute  Subtask  4,  simply  place  the  ball  anywhere  on 
the  black  background  and  enter  the  following: 

EX  SEARCH 

32.  Finally,  to  execute  Task  2,  closed  loop,  dynamic  look- 
and-move  visual  servo  control,  place  the  target  near 
the  center  of  the  camera’s  FOV  and  enter  the  following: 

EX  TRACK. TARG. DCAT 

33.  When  finished  for  the  day,  follow  the  manuals  for 
shutting  the  systems  down. 
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